{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "using VMLS\n",
    "using LinearAlgebra"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chapter 18\n",
    "# Nonlinear least squares\n",
    "### 18.1 Nonlinear equations and least squares\n",
    "### 18.2 Gauss–Newton algorithm\n",
    "**Basic Gauss–Newton algorithm.** Let’s first implement the basic Gauss–Newton method (algorithm [18.1](https://web.stanford.edu/~boyd/vmls/vmls.pdf#algorithmctr.18.1) in VMLS) in Julia. In Julia, you can pass a function as an argument to another function, so we can pass `f` (the function) and also `Df` (the derivative or Jacobian matrix) to our Gauss–Newton algorithm."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```julia\n",
    "function gauss_newton(f, Df, x1; kmax = 10)\n",
    "    x = x1\n",
    "    for k = 1:kmax\n",
    "        x = x - Df(x) \\ f(x)\n",
    "    end\n",
    "    return x\n",
    "end\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we simply run the algorithm for a fixed number of iterations `kmax`, specified\n",
    "by an optional keyword argument with default value $10$. The code does not verify\n",
    "whether the final `x` is actually a solution, and it will break down when $Df(x(k))$ has\n",
    "linearly dependent columns. This very simple implementation is only for illustrative\n",
    "purposes; the Levenberg–Marquardt algorithm described in the next section is\n",
    "better in every way.\n",
    "\n",
    "**Newton algorithm.** The Gauss–Newton algorithm reduces to the Newton algo-\n",
    "rithm when the function maps $n$-vectors to $n$-vectors, so the function above is also\n",
    "an implementation of the Newton method for solving nonlinear equations. The\n",
    "only difference with the following function is the stopping condition. In Newton’s\n",
    "method one terminates when $‖f(x^{(k)})‖$ is sufficiently small.\n",
    "                        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "newton (generic function with 1 method)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "function newton(f, Df, x1; kmax = 20, tol = 1e-6)\n",
    "    x = x1\n",
    "    fnorms = zeros(0,1)\n",
    "    for k = 1:kmax\n",
    "        fk = f(x)\n",
    "        fnorms = [fnorms; norm(fk)]\n",
    "        if norm(fk) < tol\n",
    "            break\n",
    "            end;\n",
    "        x = x - Df(x) \\ fk\n",
    "    end\n",
    "    return x, fnorms\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We added a second optional argument with the tolerance in the stopping condition on line $7$. The default value is $10−6$. We also added a second output argument `fnorms`, with the sequence $‖f(x^{(k)})‖$, so we can examine the convergence in the following examples.\n",
    "\n",
    "**Newton algorithm for n = 1.** Our first example is a scalar nonlinear equation\n",
    "$f(x) = 0$ with\n",
    "$$\n",
    "f(x) = \\frac{e^{x} − e^{−x}}{e^{x} + e^{−x}}\n",
    "$$\n",
    "(VMLS figures [18.3](https://web.stanford.edu/~boyd/vmls/vmls.pdf#figure.18.3) and [18.4](https://web.stanford.edu/~boyd/vmls/vmls.pdf#figure.18.4)). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "f(x) = (exp(x)-exp(-x)) / (exp(x)+exp(-x));\n",
    "Df(x) = 4 / (exp(x) + exp(-x))^2;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first try with $x^{(1)} = 0.95$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4.3451974324200454e-7"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x, fnorms = newton(f,Df,0.95);\n",
    "f(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5×1 Array{Float64,2}:\n",
       " 0.7397830512740042   \n",
       " 0.5941663642651942   \n",
       " 0.23011124550034218  \n",
       " 0.00867002864500575  \n",
       " 4.3451974324200454e-7"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fnorms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"600\" height=\"400\" viewBox=\"0 0 2400 1600\">\n",
       "<defs>\n",
       "  <clipPath id=\"clip4300\">\n",
       "    <rect x=\"0\" y=\"0\" width=\"2400\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<path clip-path=\"url(#clip4300)\" d=\"\n",
       "M0 1600 L2400 1600 L2400 0 L0 0  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip4301\">\n",
       "    <rect x=\"480\" y=\"0\" width=\"1681\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<path clip-path=\"url(#clip4300)\" d=\"\n",
       "M215.754 1425.62 L2352.76 1425.62 L2352.76 47.2441 L215.754 47.2441  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip4302\">\n",
       "    <rect x=\"215\" y=\"47\" width=\"2138\" height=\"1379\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polyline clip-path=\"url(#clip4302)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  276.235,1425.62 276.235,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4302)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  780.245,1425.62 780.245,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4302)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1284.25,1425.62 1284.25,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4302)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1788.26,1425.62 1788.26,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4302)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  2292.27,1425.62 2292.27,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4302)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  215.754,1386.61 2352.76,1386.61 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4302)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  215.754,1035.06 2352.76,1035.06 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4302)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  215.754,683.509 2352.76,683.509 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4302)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  215.754,331.958 2352.76,331.958 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4300)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  215.754,1425.62 2352.76,1425.62 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4300)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  215.754,1425.62 215.754,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4300)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  276.235,1425.62 276.235,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4300)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  780.245,1425.62 780.245,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4300)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1284.25,1425.62 1284.25,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4300)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1788.26,1425.62 1788.26,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4300)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  2292.27,1425.62 2292.27,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4300)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  215.754,1386.61 247.809,1386.61 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4300)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  215.754,1035.06 247.809,1035.06 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4300)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  215.754,683.509 247.809,683.509 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4300)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  215.754,331.958 247.809,331.958 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip4300)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 276.235, 1479.62)\" x=\"276.235\" y=\"1479.62\">1</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4300)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 780.245, 1479.62)\" x=\"780.245\" y=\"1479.62\">2</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4300)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 1284.25, 1479.62)\" x=\"1284.25\" y=\"1479.62\">3</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4300)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 1788.26, 1479.62)\" x=\"1788.26\" y=\"1479.62\">4</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4300)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 2292.27, 1479.62)\" x=\"2292.27\" y=\"1479.62\">5</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4300)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 191.754, 1404.11)\" x=\"191.754\" y=\"1404.11\">0.0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4300)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 191.754, 1052.56)\" x=\"191.754\" y=\"1052.56\">0.2</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4300)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 191.754, 701.009)\" x=\"191.754\" y=\"701.009\">0.4</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4300)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 191.754, 349.458)\" x=\"191.754\" y=\"349.458\">0.6</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4300)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:66px; text-anchor:middle;\" transform=\"rotate(0, 1284.25, 1559.48)\" x=\"1284.25\" y=\"1559.48\">k</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4300)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:66px; text-anchor:middle;\" transform=\"rotate(-90, 89.2861, 736.431)\" x=\"89.2861\" y=\"736.431\">|f|</text>\n",
       "</g>\n",
       "<polyline clip-path=\"url(#clip4302)\" style=\"stroke:#009af9; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  276.235,86.2547 780.245,342.213 1284.25,982.131 1788.26,1371.37 2292.27,1386.61 \n",
       "  \"/>\n",
       "<circle clip-path=\"url(#clip4302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"276.235\" cy=\"86.2547\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"276.235\" cy=\"86.2547\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip4302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"780.245\" cy=\"342.213\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"780.245\" cy=\"342.213\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip4302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1284.25\" cy=\"982.131\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1284.25\" cy=\"982.131\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip4302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1788.26\" cy=\"1371.37\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1788.26\" cy=\"1371.37\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip4302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2292.27\" cy=\"1386.61\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2292.27\" cy=\"1386.61\" r=\"14\"/>\n",
       "</svg>\n"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "using Plots\n",
    "plot(fnorms, shape=:circle, legend = false, xlabel = \"k\",ylabel = \"|f|\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Figure 18.1**<a ind=\"#fig1\"></a> The first iterations in the Newton algorithm for solving $f(x) = 0$ for starting point $x(1) = 0.95$. \n",
    "\n",
    "The method converges very quickly, as can also be seen in figure [18.1](#fig1). However it does not converge for a slightly larger starting point $x(1) = 1.15$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NaN"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x, fnorms = newton(f,Df,1.15);\n",
    "f(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5-element Array{Float64,1}:\n",
       "   0.8177540779702877\n",
       "   0.8664056534177534\n",
       "   0.9735568532451108\n",
       "   0.9999999999999906\n",
       " NaN                 "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fnorms[1:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 18.3 Levenberg–Marquardt algorithm \n",
    "The Gauss–Newton algorithm can fail if the derivative matrix does not have independent columns. It also does not guarantee that $‖f(x^{(k)})‖$ decreases in each iteration. Both of these shortcomings are addressed in the Levenberg–Marquardt algorithm. Below is a Julia implementation of algorithm [18.3](https://web.stanford.edu/~boyd/vmls/vmls.pdf#algorithmctr.18.3) in VMLS. This function is included in the `Vmls` package."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```julia\n",
    "function levenberg_marquardt(f, Df, x1, lambda1; kmax=100, tol=1e-6)\n",
    "    n = length(x1)\n",
    "    x = x1\n",
    "    lambda = lambda1\n",
    "    objectives = zeros(0,1)\n",
    "    residuals = zeros(0,1)\n",
    "    for k = 1:kmax\n",
    "        fk = f(x)\n",
    "        Dfk = Df(x)\n",
    "        objectives = [objectives; norm(fk)^2]\n",
    "        residuals = [residuals; norm(2*Dfk'*fk)]\n",
    "        if norm(2*Dfk'*fk) < tol\n",
    "            break\n",
    "            end;\n",
    "        xt = x - [ Dfk; sqrt(lambda)*eye(n) ] \\ [ fk; zeros(n) ]\n",
    "        if norm(f(xt)) < norm(fk)\n",
    "            lambda = 0.8*lambda\n",
    "            x = xt\n",
    "        else\n",
    "            lambda = 2.0*lambda\n",
    "        end\n",
    "    end\n",
    "    return x, Dict([ (\"objectives\", objectives),\n",
    "            (\"residuals\", residuals)])\n",
    "end\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Line $12$ (`if norm(2*Dfk'*fk)`) is the second stopping criterion suggested on page [393](https://web.stanford.edu/~boyd/vmls/vmls.pdf#section*.408) of VMLS, and checks whether the optimality condition ([18.3](https://web.stanford.edu/~boyd/vmls/vmls.pdf#equation.18.1.3)) is approximately satisfied. The default tolerance $10^{−6}$ can vary with the scale of the problem and the desired accuracy. Keep in mind that the optimality condition ([18.3](https://web.stanford.edu/~boyd/vmls/vmls.pdf#equation.18.1.3)) is a necessary condition and does not guarantee that the solution minimizes the nonlinear least squares objective $‖f(x)‖^2$. The code limits the number of iterations to `kmax`, after which it is assumed that the algorithm is failing to converge. \n",
    "\n",
    "The function returns a dictionary with information about the sequence of iterates, including the value of $‖f(x^{(k)}‖^2$ and $‖Df(x^{(k)})^T f(x^{(k)})‖$ at each iteration. \n",
    "\n",
    "Nonlinear equation. We apply the algorithm to the scalar function ([18.1](#fig1)) with the starting point $x(1) = 1.15$.\n",
    "            \n",
    "The result is shown in figure [18.2](#fig2)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"600\" height=\"400\" viewBox=\"0 0 2400 1600\">\n",
       "<defs>\n",
       "  <clipPath id=\"clip4700\">\n",
       "    <rect x=\"0\" y=\"0\" width=\"2400\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<path clip-path=\"url(#clip4700)\" d=\"\n",
       "M0 1600 L2400 1600 L2400 0 L0 0  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip4701\">\n",
       "    <rect x=\"480\" y=\"0\" width=\"1681\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<path clip-path=\"url(#clip4700)\" d=\"\n",
       "M215.754 1425.62 L2352.76 1425.62 L2352.76 47.2441 L215.754 47.2441  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip4702\">\n",
       "    <rect x=\"215\" y=\"47\" width=\"2138\" height=\"1379\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polyline clip-path=\"url(#clip4702)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  500.24,1425.62 500.24,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4702)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  948.248,1425.62 948.248,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4702)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1396.26,1425.62 1396.26,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4702)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1844.27,1425.62 1844.27,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4702)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  2292.27,1425.62 2292.27,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4702)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  215.754,1386.66 2352.76,1386.66 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4702)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  215.754,1068.62 2352.76,1068.62 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4702)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  215.754,750.573 2352.76,750.573 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4702)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  215.754,432.53 2352.76,432.53 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4702)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  215.754,114.487 2352.76,114.487 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4700)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  215.754,1425.62 2352.76,1425.62 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4700)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  215.754,1425.62 215.754,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4700)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  500.24,1425.62 500.24,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4700)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  948.248,1425.62 948.248,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4700)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1396.26,1425.62 1396.26,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4700)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1844.27,1425.62 1844.27,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4700)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  2292.27,1425.62 2292.27,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4700)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  215.754,1386.66 247.809,1386.66 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4700)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  215.754,1068.62 247.809,1068.62 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4700)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  215.754,750.573 247.809,750.573 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4700)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  215.754,432.53 247.809,432.53 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip4700)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  215.754,114.487 247.809,114.487 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip4700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 500.24, 1479.62)\" x=\"500.24\" y=\"1479.62\">2</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 948.248, 1479.62)\" x=\"948.248\" y=\"1479.62\">4</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 1396.26, 1479.62)\" x=\"1396.26\" y=\"1479.62\">6</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 1844.27, 1479.62)\" x=\"1844.27\" y=\"1479.62\">8</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 2292.27, 1479.62)\" x=\"2292.27\" y=\"1479.62\">10</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 191.754, 1404.16)\" x=\"191.754\" y=\"1404.16\">0.0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 191.754, 1086.12)\" x=\"191.754\" y=\"1086.12\">0.2</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 191.754, 768.073)\" x=\"191.754\" y=\"768.073\">0.4</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 191.754, 450.03)\" x=\"191.754\" y=\"450.03\">0.6</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 191.754, 131.987)\" x=\"191.754\" y=\"131.987\">0.8</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:66px; text-anchor:middle;\" transform=\"rotate(0, 1284.25, 1559.48)\" x=\"1284.25\" y=\"1559.48\">k</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip4700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:66px; text-anchor:middle;\" transform=\"rotate(-90, 89.2861, 736.431)\" x=\"89.2861\" y=\"736.431\">|f|</text>\n",
       "</g>\n",
       "<polyline clip-path=\"url(#clip4702)\" style=\"stroke:#009af9; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  276.235,86.2547 500.24,243.053 724.244,567.367 948.248,998.742 1172.25,1252.32 1396.26,1347.59 1620.26,1377.02 1844.27,1384.66 2068.27,1386.31 2292.27,1386.61 \n",
       "  \n",
       "  \"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"276.235\" cy=\"86.2547\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"276.235\" cy=\"86.2547\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"500.24\" cy=\"243.053\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"500.24\" cy=\"243.053\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"724.244\" cy=\"567.367\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"724.244\" cy=\"567.367\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"948.248\" cy=\"998.742\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"948.248\" cy=\"998.742\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1172.25\" cy=\"1252.32\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1172.25\" cy=\"1252.32\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1396.26\" cy=\"1347.59\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1396.26\" cy=\"1347.59\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1620.26\" cy=\"1377.02\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1620.26\" cy=\"1377.02\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1844.27\" cy=\"1384.66\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1844.27\" cy=\"1384.66\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2068.27\" cy=\"1386.31\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2068.27\" cy=\"1386.31\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2292.27\" cy=\"1386.61\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip4702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2292.27\" cy=\"1386.61\" r=\"14\"/>\n",
       "</svg>\n"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f(x) = (exp.(x) - exp.(-x)) / (exp.(x) + exp.(-x));\n",
    "Df(x) = 4 ./ (exp.(x) + exp.(-x)).^2;\n",
    "x, history = levenberg_marquardt(f, Df, [1.15], 1.0);\n",
    "plot(sqrt.(history[\"objectives\"][1:10]), shape = :circle,\n",
    "    legend = false, xlabel = \"k\", ylabel = \"|f|\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Figure 18.<a ind=\"#fig2\"></a>2** Values of $|f(x(k))|$ versus the iteration number $k$ for the Levenberg–Marquardt algorithm applied to \n",
    "$$\n",
    "f(x) = \\frac{(exp(x) − exp(−x))}{(exp(x)+exp(−x))}.\n",
    "$$ \n",
    "\n",
    "The starting point is $x^{(1)} = 1.15$ and $λ^{(1)} = 1$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we defined $x^{(1)}$ as the array `[1.15]`, and use dot-operations in the definitions of `f` and `Df` to ensure that these functions work with vector arguments. This is important because Julia distinguishes between scalars and $1$-vectors. If we call the `levenberg_marquardt` function with a scalar argument `x1`, line $15$ will raise an `error`, because Julia does not accept subtractions of scalars and $1$-vectors.\n",
    "\n",
    "**Equilibrium prices.** We solve a nonlinear equation $f(p) = 0$ with two variables, where \n",
    "\n",
    "$$\n",
    "f(p) = exp(E^s log p+ s^{nom})− exp(E^d log p+ d^{nom}). \n",
    "$$\n",
    "\n",
    "Here $exp$ and $log$ are interpreted as element-wise vector operations. The problem parameters are $s^{nom} = (2.2, 0.3), d^{nom} = (3.1, 2.2)$,\n",
    "\n",
    "$$\n",
    "E^s =\n",
    "\\begin{bmatrix}\n",
    "0.5 & −0.3\\\\\n",
    "−0.15 &  0.8\n",
    "\\end{bmatrix}\n",
    ", E^d =\n",
    "\\begin{bmatrix}\n",
    "−0.5 & 0.2\\\\\n",
    "0 & −0.5\n",
    "\\end{bmatrix}.\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2-element Array{Float64,1}:\n",
       " 5.644108533416072\n",
       " 5.265754930531486"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "snom = [2.2, 0.3];\n",
    "dnom = [3.1, 2.2];\n",
    "Es = [0.5 -.3; -0.15 0.8];\n",
    "Ed = [-0.5 0.2; -0.00 -0.5];\n",
    "f(p) = exp.(Es * log.(p) + snom) - exp.(Ed * log.(p) + dnom);\n",
    "function Df(p)\n",
    "    S = exp.(Es * log.(p) + snom);\n",
    "    D = exp.(Ed * log.(p) + dnom);\n",
    "    return [ S[1]*Es[1,1]/p[1] S[1]*Es[1,2]/p[2]; \n",
    "        S[2]*Es[2,1]/p[1] S[2]*Es[2,2]/p[2] ] - [ D[1]*Ed[1,1]/p[1] D[1]*Ed[1,2]/p[2];\n",
    "        D[2]*Ed[2,1]/p[1] D[2]*Ed[2,2]/p[2] ];\n",
    "end;\n",
    "p, history = levenberg_marquardt(f, Df, [3, 9], 1);\n",
    "p"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"600\" height=\"400\" viewBox=\"0 0 2400 1600\">\n",
       "<defs>\n",
       "  <clipPath id=\"clip5100\">\n",
       "    <rect x=\"0\" y=\"0\" width=\"2400\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<path clip-path=\"url(#clip5100)\" d=\"\n",
       "M0 1600 L2400 1600 L2400 0 L0 0  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip5101\">\n",
       "    <rect x=\"480\" y=\"0\" width=\"1681\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<path clip-path=\"url(#clip5100)\" d=\"\n",
       "M229.135 1425.62 L2352.76 1425.62 L2352.76 47.2441 L229.135 47.2441  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip5102\">\n",
       "    <rect x=\"229\" y=\"47\" width=\"2125\" height=\"1379\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polyline clip-path=\"url(#clip5102)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  520.401,1425.62 520.401,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5102)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  905.673,1425.62 905.673,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5102)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1290.95,1425.62 1290.95,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5102)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1676.22,1425.62 1676.22,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5102)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  2061.49,1425.62 2061.49,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5102)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  229.135,1386.61 2352.76,1386.61 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5102)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  229.135,960.206 2352.76,960.206 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5102)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  229.135,533.804 2352.76,533.804 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5102)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  229.135,107.402 2352.76,107.402 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5100)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  229.135,1425.62 2352.76,1425.62 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5100)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  229.135,1425.62 229.135,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5100)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  520.401,1425.62 520.401,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5100)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  905.673,1425.62 905.673,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5100)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1290.95,1425.62 1290.95,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5100)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1676.22,1425.62 1676.22,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5100)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  2061.49,1425.62 2061.49,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5100)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  229.135,1386.61 260.99,1386.61 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5100)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  229.135,960.206 260.99,960.206 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5100)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  229.135,533.804 260.99,533.804 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5100)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  229.135,107.402 260.99,107.402 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip5100)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 520.401, 1479.62)\" x=\"520.401\" y=\"1479.62\">2.5</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5100)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 905.673, 1479.62)\" x=\"905.673\" y=\"1479.62\">5.0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5100)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 1290.95, 1479.62)\" x=\"1290.95\" y=\"1479.62\">7.5</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5100)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 1676.22, 1479.62)\" x=\"1676.22\" y=\"1479.62\">10.0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5100)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 2061.49, 1479.62)\" x=\"2061.49\" y=\"1479.62\">12.5</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5100)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 205.135, 1404.11)\" x=\"205.135\" y=\"1404.11\">0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5100)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 205.135, 977.706)\" x=\"205.135\" y=\"977.706\">50</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5100)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 205.135, 551.304)\" x=\"205.135\" y=\"551.304\">100</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5100)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 205.135, 124.902)\" x=\"205.135\" y=\"124.902\">150</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5100)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:66px; text-anchor:middle;\" transform=\"rotate(0, 1290.95, 1559.48)\" x=\"1290.95\" y=\"1559.48\">k</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5100)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:66px; text-anchor:middle;\" transform=\"rotate(-90, 89.2861, 736.431)\" x=\"89.2861\" y=\"736.431\">Objective</text>\n",
       "</g>\n",
       "<polyline clip-path=\"url(#clip5102)\" style=\"stroke:#009af9; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  289.238,86.2547 443.347,1265.38 597.456,1378.96 751.564,1384.83 905.673,1386.21 1059.78,1386.54 1213.89,1386.6 1368,1386.61 1522.11,1386.61 1676.22,1386.61 \n",
       "  1830.33,1386.61 1984.44,1386.61 2138.54,1386.61 2292.65,1386.61 \n",
       "  \"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"289.238\" cy=\"86.2547\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"289.238\" cy=\"86.2547\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"443.347\" cy=\"1265.38\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"443.347\" cy=\"1265.38\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"597.456\" cy=\"1378.96\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"597.456\" cy=\"1378.96\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"751.564\" cy=\"1384.83\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"751.564\" cy=\"1384.83\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"905.673\" cy=\"1386.21\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"905.673\" cy=\"1386.21\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1059.78\" cy=\"1386.54\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1059.78\" cy=\"1386.54\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1213.89\" cy=\"1386.6\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1213.89\" cy=\"1386.6\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1368\" cy=\"1386.61\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1368\" cy=\"1386.61\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1522.11\" cy=\"1386.61\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1522.11\" cy=\"1386.61\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1676.22\" cy=\"1386.61\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1676.22\" cy=\"1386.61\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1830.33\" cy=\"1386.61\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1830.33\" cy=\"1386.61\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1984.44\" cy=\"1386.61\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1984.44\" cy=\"1386.61\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2138.54\" cy=\"1386.61\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2138.54\" cy=\"1386.61\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2292.65\" cy=\"1386.61\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5102)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2292.65\" cy=\"1386.61\" r=\"14\"/>\n",
       "</svg>\n"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "using Plots\n",
    "plot(history[\"objectives\"], shape = :circle, legend =false, xlabel = \"k\", ylabel = \"Objective\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Figure 18.<a ind=\"#fig\"></a>3** Cost function $‖f(p^{(k)}‖^2$ versus iteration number $k$ for the example of equation ([18.2](#fig2)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Location from range measurements.** The next example is the location from range measurements problem on page [396](https://web.stanford.edu/~boyd/vmls/vmls.pdf#section*.415) in VMLS. The positions of the $m = 5$ points $a_i$ are given as rows in a $5 × 2$ matrix $A$. The measurements are given in a $5$-vector `rhos`. To simplify the code for the functions `f(x)` and `Df(x)` we add a function `dist(x)` that computes the vector of distances ($‖x− a_1‖, . . . , ‖x− a_m‖$)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The expression for the derivative is\n",
    "\\begin{equation}\n",
    "Df(x) = \n",
    "\\begin{bmatrix}\n",
    "   \\frac{x_1 - (a_{1})_{1}}{||x - a_1||} & \\frac{x_2 - (a_{1})_{2}}{||x - a_1||} & \\\\\n",
    "    \\vdots & \\vdots \\\\\n",
    "    \\frac{x_1 - (a_{m})_{1}}{||x - a_m||} & \\frac{x_2 - (a_{m})_{2}}{||x - a_m||}\n",
    "\\end{bmatrix}\n",
    ".\n",
    "\\end{equation}\n",
    "\n",
    "This can be evaluated as the product of a diagonal matrix with diagonal entries\n",
    "$\\frac{1}{‖x− a_i‖}$ and the $5 × 2$ matrix with $i, j$ entry $(x − a_i)_j$ .\n",
    "\n",
    "We run the `Levenberg–Marquardt` method for three starting points and $λ^{(1)} =\n",
    "0.1$. The plot is shown in figure [18.4](#fig4)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2-element Array{Float64,1}:\n",
       " 1.1824859803827907\n",
       " 0.8242289367900364"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Five locations ai in a 5x2 matrix.\n",
    "A = [ 1.8 2.5; 2.0 1.7; 1.5 1.5; 1.5 2.0; 2.5 1.5 ];\n",
    "# Vector of measured distances to five locations.\n",
    "rhos = [ 1.87288, 1.23950, 0.53672, 1.29273, 1.49353 ];\n",
    "# dist(x) returns a 5-vector with the distances ||x-ai||.\n",
    "dist(x) = sqrt.( (x[1] .- A[:,1]).^2 + (x[2] .- A[:,2]).^2 );\n",
    "# f(x) returns the five residuals.\n",
    "f(x) = dist(x) - rhos;\n",
    "# Df(x) is the 5x2 derivative.\n",
    "Df(x) = diagonal(1 ./ dist(x)) * [ (x[1] .- A[:,1]) (x[2] .- A[:,2]) ];\n",
    "# Solve with starting point (1.8,3.5) and lambda = 0.1.\n",
    "x1, history1 = levenberg_marquardt(f, Df, [1.8, 3.5], 0.1);\n",
    "x1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2-element Array{Float64,1}:\n",
       " 1.1824857942435818\n",
       " 0.8242289466379732"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Starting point (3.0,1.5).\n",
    "x2, history2 = levenberg_marquardt(f, Df, [3.0, 1.5], 0.1);\n",
    "x2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2-element Array{Float64,1}:\n",
       " 2.9852664103617954\n",
       " 2.1215768036188956"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Starting point (2.2,3.5).\n",
    "x3, history3 = levenberg_marquardt(f, Df, [2.2, 3.5], 0.1);\n",
    "x3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"600\" height=\"400\" viewBox=\"0 0 2400 1600\">\n",
       "<defs>\n",
       "  <clipPath id=\"clip5500\">\n",
       "    <rect x=\"0\" y=\"0\" width=\"2400\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<path clip-path=\"url(#clip5500)\" d=\"\n",
       "M0 1600 L2400 1600 L2400 0 L0 0  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip5501\">\n",
       "    <rect x=\"480\" y=\"0\" width=\"1681\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<path clip-path=\"url(#clip5500)\" d=\"\n",
       "M175.611 1425.62 L2352.76 1425.62 L2352.76 47.2441 L175.611 47.2441  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip5502\">\n",
       "    <rect x=\"175\" y=\"47\" width=\"2178\" height=\"1379\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polyline clip-path=\"url(#clip5502)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  465.44,1425.62 465.44,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5502)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  921.865,1425.62 921.865,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5502)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1378.29,1425.62 1378.29,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5502)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1834.71,1425.62 1834.71,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5502)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  2291.14,1425.62 2291.14,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5502)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  175.611,1406.75 2352.76,1406.75 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5502)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  175.611,1065.95 2352.76,1065.95 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5502)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  175.611,725.152 2352.76,725.152 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5502)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  175.611,384.351 2352.76,384.351 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5500)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  175.611,1425.62 2352.76,1425.62 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5500)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  175.611,1425.62 175.611,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5500)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  465.44,1425.62 465.44,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5500)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  921.865,1425.62 921.865,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5500)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1378.29,1425.62 1378.29,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5500)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1834.71,1425.62 1834.71,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5500)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  2291.14,1425.62 2291.14,1404.94 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5500)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  175.611,1406.75 208.268,1406.75 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5500)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  175.611,1065.95 208.268,1065.95 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5500)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  175.611,725.152 208.268,725.152 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5500)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  175.611,384.351 208.268,384.351 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip5500)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 465.44, 1479.62)\" x=\"465.44\" y=\"1479.62\">2</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5500)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 921.865, 1479.62)\" x=\"921.865\" y=\"1479.62\">4</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5500)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 1378.29, 1479.62)\" x=\"1378.29\" y=\"1479.62\">6</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5500)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 1834.71, 1479.62)\" x=\"1834.71\" y=\"1479.62\">8</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5500)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 2291.14, 1479.62)\" x=\"2291.14\" y=\"1479.62\">10</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5500)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 151.611, 1424.25)\" x=\"151.611\" y=\"1424.25\">0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5500)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 151.611, 1083.45)\" x=\"151.611\" y=\"1083.45\">1</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5500)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 151.611, 742.652)\" x=\"151.611\" y=\"742.652\">2</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5500)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 151.611, 401.851)\" x=\"151.611\" y=\"401.851\">3</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5500)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:66px; text-anchor:middle;\" transform=\"rotate(0, 1264.18, 1559.48)\" x=\"1264.18\" y=\"1559.48\">k</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5500)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:66px; text-anchor:middle;\" transform=\"rotate(-90, 89.2861, 736.431)\" x=\"89.2861\" y=\"736.431\">Objective</text>\n",
       "</g>\n",
       "<polyline clip-path=\"url(#clip5502)\" style=\"stroke:#009af9; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  237.228,131.097 465.44,415.852 693.652,512.964 921.865,512.964 1150.08,577.803 1378.29,598.891 1606.5,1317.2 1834.71,1386.47 2062.93,1386.61 2291.14,1386.61 \n",
       "  \n",
       "  \"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"237.228\" cy=\"131.097\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"237.228\" cy=\"131.097\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"465.44\" cy=\"415.852\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"465.44\" cy=\"415.852\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"693.652\" cy=\"512.964\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"693.652\" cy=\"512.964\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"921.865\" cy=\"512.964\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"921.865\" cy=\"512.964\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1150.08\" cy=\"577.803\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1150.08\" cy=\"577.803\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1378.29\" cy=\"598.891\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1378.29\" cy=\"598.891\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1606.5\" cy=\"1317.2\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1606.5\" cy=\"1317.2\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1834.71\" cy=\"1386.47\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1834.71\" cy=\"1386.47\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2062.93\" cy=\"1386.61\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2062.93\" cy=\"1386.61\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2291.14\" cy=\"1386.61\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2291.14\" cy=\"1386.61\" r=\"14\"/>\n",
       "<polyline clip-path=\"url(#clip5502)\" style=\"stroke:#e26f46; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  237.228,676.393 465.44,714.687 693.652,714.687 921.865,860.225 1150.08,860.225 1378.29,1118.84 1606.5,1374.05 1834.71,1386.57 2062.93,1386.61 2291.14,1386.61 \n",
       "  \n",
       "  \"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"237.228\" cy=\"676.393\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"237.228\" cy=\"676.393\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"465.44\" cy=\"714.687\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"465.44\" cy=\"714.687\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"693.652\" cy=\"714.687\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"693.652\" cy=\"714.687\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"921.865\" cy=\"860.225\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"921.865\" cy=\"860.225\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1150.08\" cy=\"860.225\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"1150.08\" cy=\"860.225\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1378.29\" cy=\"1118.84\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"1378.29\" cy=\"1118.84\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1606.5\" cy=\"1374.05\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"1606.5\" cy=\"1374.05\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1834.71\" cy=\"1386.57\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"1834.71\" cy=\"1386.57\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2062.93\" cy=\"1386.61\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"2062.93\" cy=\"1386.61\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2291.14\" cy=\"1386.61\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"2291.14\" cy=\"1386.61\" r=\"14\"/>\n",
       "<polyline clip-path=\"url(#clip5502)\" style=\"stroke:#3da44d; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  237.228,86.2547 465.44,513.44 693.652,628.756 921.865,682.213 1150.08,686.762 1378.29,687.093 1606.5,687.145 1834.71,687.156 2062.93,687.158 2291.14,687.159 \n",
       "  \n",
       "  \"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"237.228\" cy=\"86.2547\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#3da44d; stroke:none; fill-opacity:1\" cx=\"237.228\" cy=\"86.2547\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"465.44\" cy=\"513.44\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#3da44d; stroke:none; fill-opacity:1\" cx=\"465.44\" cy=\"513.44\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"693.652\" cy=\"628.756\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#3da44d; stroke:none; fill-opacity:1\" cx=\"693.652\" cy=\"628.756\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"921.865\" cy=\"682.213\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#3da44d; stroke:none; fill-opacity:1\" cx=\"921.865\" cy=\"682.213\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1150.08\" cy=\"686.762\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#3da44d; stroke:none; fill-opacity:1\" cx=\"1150.08\" cy=\"686.762\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1378.29\" cy=\"687.093\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#3da44d; stroke:none; fill-opacity:1\" cx=\"1378.29\" cy=\"687.093\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1606.5\" cy=\"687.145\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#3da44d; stroke:none; fill-opacity:1\" cx=\"1606.5\" cy=\"687.145\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1834.71\" cy=\"687.156\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#3da44d; stroke:none; fill-opacity:1\" cx=\"1834.71\" cy=\"687.156\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2062.93\" cy=\"687.158\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#3da44d; stroke:none; fill-opacity:1\" cx=\"2062.93\" cy=\"687.158\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2291.14\" cy=\"687.159\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5502)\" style=\"fill:#3da44d; stroke:none; fill-opacity:1\" cx=\"2291.14\" cy=\"687.159\" r=\"14\"/>\n",
       "<path clip-path=\"url(#clip5500)\" d=\"\n",
       "M1717.01 372.684 L2280.76 372.684 L2280.76 130.764 L1717.01 130.764  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip5500)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1717.01,372.684 2280.76,372.684 2280.76,130.764 1717.01,130.764 1717.01,372.684 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5500)\" style=\"stroke:#009af9; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1741.01,191.244 1885.01,191.244 \n",
       "  \"/>\n",
       "<circle clip-path=\"url(#clip5500)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1825.01\" cy=\"191.244\" r=\"25\"/>\n",
       "<circle clip-path=\"url(#clip5500)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1825.01\" cy=\"191.244\" r=\"21\"/>\n",
       "<g clip-path=\"url(#clip5500)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 1909.01, 208.744)\" x=\"1909.01\" y=\"208.744\">Starting point 1</text>\n",
       "</g>\n",
       "<polyline clip-path=\"url(#clip5500)\" style=\"stroke:#e26f46; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1741.01,251.724 1885.01,251.724 \n",
       "  \"/>\n",
       "<circle clip-path=\"url(#clip5500)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1825.01\" cy=\"251.724\" r=\"25\"/>\n",
       "<circle clip-path=\"url(#clip5500)\" style=\"fill:#e26f46; stroke:none; fill-opacity:1\" cx=\"1825.01\" cy=\"251.724\" r=\"21\"/>\n",
       "<g clip-path=\"url(#clip5500)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 1909.01, 269.224)\" x=\"1909.01\" y=\"269.224\">Starting point 2</text>\n",
       "</g>\n",
       "<polyline clip-path=\"url(#clip5500)\" style=\"stroke:#3da44d; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1741.01,312.204 1885.01,312.204 \n",
       "  \"/>\n",
       "<circle clip-path=\"url(#clip5500)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1825.01\" cy=\"312.204\" r=\"25\"/>\n",
       "<circle clip-path=\"url(#clip5500)\" style=\"fill:#3da44d; stroke:none; fill-opacity:1\" cx=\"1825.01\" cy=\"312.204\" r=\"21\"/>\n",
       "<g clip-path=\"url(#clip5500)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 1909.01, 329.704)\" x=\"1909.01\" y=\"329.704\">Starting point 3</text>\n",
       "</g>\n",
       "</svg>\n"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "using Plots\n",
    "plot(history1[\"objectives\"][1:10], shape = :circle,\n",
    "label = \"Starting point 1\")\n",
    "plot!(history2[\"objectives\"][1:10], shape = :circle,\n",
    "label = \"Starting point 2\")\n",
    "plot!(history3[\"objectives\"][1:10], shape = :circle,\n",
    "label = \"Starting point 3\")\n",
    "plot!(xlabel = \"k\", ylabel = \"Objective\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Figure 18.<a ind=\"#fig4\"></a>4** Cost function $‖f(x^{(k)})‖^2$ versus iteration number $k$ for the three starting points in the location from range measurements example.\n",
    "\n",
    "### 18.4 Nonlinear model fitting\n",
    "**Example.** We fit a model\n",
    "$$\n",
    "f̂(x; θ) = θ_1e^{θ{_2x}} cos(θ_3x+ θ_4)\n",
    "$$\n",
    "to $N = 60$ data points. We first generate the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"600\" height=\"400\" viewBox=\"0 0 2400 1600\">\n",
       "<defs>\n",
       "  <clipPath id=\"clip5900\">\n",
       "    <rect x=\"0\" y=\"0\" width=\"2400\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<path clip-path=\"url(#clip5900)\" d=\"\n",
       "M0 1600 L2400 1600 L2400 0 L0 0  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip5901\">\n",
       "    <rect x=\"480\" y=\"0\" width=\"1681\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<path clip-path=\"url(#clip5900)\" d=\"\n",
       "M182.008 1487.47 L2352.76 1487.47 L2352.76 47.2441 L182.008 47.2441  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip5902\">\n",
       "    <rect x=\"182\" y=\"47\" width=\"2172\" height=\"1441\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polyline clip-path=\"url(#clip5902)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  236.938,1487.47 236.938,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5902)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  766.783,1487.47 766.783,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5902)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1296.63,1487.47 1296.63,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5902)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1826.47,1487.47 1826.47,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5902)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  182.008,1227.32 2352.76,1227.32 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5902)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  182.008,944.153 2352.76,944.153 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5902)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  182.008,660.989 2352.76,660.989 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5902)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  182.008,377.826 2352.76,377.826 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5902)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  182.008,94.6629 2352.76,94.6629 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5900)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  182.008,1487.47 2352.76,1487.47 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5900)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  182.008,1487.47 182.008,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5900)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  236.938,1487.47 236.938,1465.87 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5900)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  766.783,1487.47 766.783,1465.87 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5900)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1296.63,1487.47 1296.63,1465.87 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5900)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1826.47,1487.47 1826.47,1465.87 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5900)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  182.008,1227.32 214.569,1227.32 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5900)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  182.008,944.153 214.569,944.153 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5900)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  182.008,660.989 214.569,660.989 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5900)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  182.008,377.826 214.569,377.826 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip5900)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  182.008,94.6629 214.569,94.6629 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip5900)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 236.938, 1541.47)\" x=\"236.938\" y=\"1541.47\">0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5900)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 766.783, 1541.47)\" x=\"766.783\" y=\"1541.47\">5</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5900)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 1296.63, 1541.47)\" x=\"1296.63\" y=\"1541.47\">10</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5900)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 1826.47, 1541.47)\" x=\"1826.47\" y=\"1541.47\">15</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5900)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 158.008, 1244.82)\" x=\"158.008\" y=\"1244.82\">-0.6</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5900)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 158.008, 961.653)\" x=\"158.008\" y=\"961.653\">-0.3</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5900)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 158.008, 678.489)\" x=\"158.008\" y=\"678.489\">0.0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5900)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 158.008, 395.326)\" x=\"158.008\" y=\"395.326\">0.3</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip5900)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 158.008, 112.163)\" x=\"158.008\" y=\"112.163\">0.6</text>\n",
       "</g>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"427.683\" cy=\"1287.85\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"427.683\" cy=\"1287.85\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"323.633\" cy=\"1109.54\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"323.633\" cy=\"1109.54\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"280.899\" cy=\"681.087\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"280.899\" cy=\"681.087\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"555.966\" cy=\"569.475\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"555.966\" cy=\"569.475\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"491.652\" cy=\"947.296\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"491.652\" cy=\"947.296\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"247.715\" cy=\"336.899\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"247.715\" cy=\"336.899\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"403.467\" cy=\"1446.71\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"403.467\" cy=\"1446.71\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"731.981\" cy=\"293.39\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"731.981\" cy=\"293.39\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"669.382\" cy=\"88.0053\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"669.382\" cy=\"88.0053\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"290.264\" cy=\"694.306\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"290.264\" cy=\"694.306\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"278.783\" cy=\"634.987\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"278.783\" cy=\"634.987\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"243.444\" cy=\"362.726\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"243.444\" cy=\"362.726\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"668.465\" cy=\"102.018\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"668.465\" cy=\"102.018\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"385.826\" cy=\"1317.89\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"385.826\" cy=\"1317.89\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"422.101\" cy=\"1246.81\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"422.101\" cy=\"1246.81\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"418.111\" cy=\"1327.92\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"418.111\" cy=\"1327.92\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"657.6\" cy=\"302.03\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"657.6\" cy=\"302.03\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"399.044\" cy=\"1176.08\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"399.044\" cy=\"1176.08\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"602.631\" cy=\"412.295\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"602.631\" cy=\"412.295\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"575.474\" cy=\"534.183\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"575.474\" cy=\"534.183\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"584.245\" cy=\"506.347\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"584.245\" cy=\"506.347\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"740.477\" cy=\"311.834\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"740.477\" cy=\"311.834\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"463.776\" cy=\"1150.91\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"463.776\" cy=\"1150.91\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"613.641\" cy=\"480.765\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"613.641\" cy=\"480.765\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"276.093\" cy=\"613.168\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"276.093\" cy=\"613.168\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"556.889\" cy=\"607.479\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"556.889\" cy=\"607.479\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"352.928\" cy=\"1261.31\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"352.928\" cy=\"1261.31\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"343.302\" cy=\"1193.01\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"343.302\" cy=\"1193.01\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"456.206\" cy=\"1120.69\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"456.206\" cy=\"1120.69\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"529.402\" cy=\"788.941\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"529.402\" cy=\"788.941\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1931.65\" cy=\"704.636\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1931.65\" cy=\"704.636\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1195.58\" cy=\"472.336\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1195.58\" cy=\"472.336\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1115.36\" cy=\"562.752\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1115.36\" cy=\"562.752\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1540.54\" cy=\"696.464\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1540.54\" cy=\"696.464\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2200.06\" cy=\"637.882\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2200.06\" cy=\"637.882\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1399.88\" cy=\"745.854\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1399.88\" cy=\"745.854\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"981.644\" cy=\"889.588\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"981.644\" cy=\"889.588\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2170.9\" cy=\"661.087\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2170.9\" cy=\"661.087\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2054.63\" cy=\"681.882\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2054.63\" cy=\"681.882\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1942.88\" cy=\"678.432\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1942.88\" cy=\"678.432\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1840.08\" cy=\"653.29\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1840.08\" cy=\"653.29\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1071.79\" cy=\"688.688\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1071.79\" cy=\"688.688\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"988.823\" cy=\"854.647\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"988.823\" cy=\"854.647\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1465.35\" cy=\"747.768\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1465.35\" cy=\"747.768\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2291.32\" cy=\"653.642\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2291.32\" cy=\"653.642\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"927.374\" cy=\"887.633\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"927.374\" cy=\"887.633\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"839.498\" cy=\"772.347\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"839.498\" cy=\"772.347\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"927.547\" cy=\"946.46\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"927.547\" cy=\"946.46\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1481.75\" cy=\"744.457\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1481.75\" cy=\"744.457\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1948.22\" cy=\"692.139\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1948.22\" cy=\"692.139\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1725.98\" cy=\"611.962\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1725.98\" cy=\"611.962\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1505.81\" cy=\"779.182\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1505.81\" cy=\"779.182\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1337.09\" cy=\"651.954\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1337.09\" cy=\"651.954\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"802.255\" cy=\"647.027\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"802.255\" cy=\"647.027\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1202.27\" cy=\"524.729\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1202.27\" cy=\"524.729\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1481.66\" cy=\"757.868\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1481.66\" cy=\"757.868\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1055.22\" cy=\"714.25\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1055.22\" cy=\"714.25\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1062.39\" cy=\"721.34\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1062.39\" cy=\"721.34\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1048.85\" cy=\"733.997\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1048.85\" cy=\"733.997\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1023.53\" cy=\"796.46\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip5902)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1023.53\" cy=\"796.46\" r=\"14\"/>\n",
       "</svg>\n"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Use these parameters to generate data.\n",
    "theta_ex = [1, -0.2, 2*pi/5, pi/3];\n",
    "# Choose 60 points x between 0 and 20.\n",
    "M = 30;\n",
    "xd = [5*rand(M); 5 .+ 15*rand(M)];\n",
    "# Evaluate function at these points.\n",
    "yd = theta_ex[1] * exp.(theta_ex[2]*xd) .* cos.(theta_ex[3] * xd .+ theta_ex[4])\n",
    "# Create a random perturbation of yd.\n",
    "N = length(xd);\n",
    "yd = yd .* (1 .+ 0.2*randn(N)) .+ 0.015 * randn(N);\n",
    "# Plot data points.\n",
    "using Plots\n",
    "scatter(xd, yd, legend=false)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The $60$ points are shown in figure [18.5](#fig5). We now run our Levenberg–Marquardt code with starting point $θ(1) = (1, 0, 1, 0)$ and $λ(1) = 1$. The fitted model is shown in figure [18.5](#fig5)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4-element Array{Float64,1}:\n",
       "  0.9323205639084152 \n",
       " -0.17994464859473006\n",
       "  1.2430972267595168 \n",
       "  1.0761484340773477 "
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f(theta) = theta[1] * exp.(theta[2]*xd) .* cos.(theta[3] * xd .+ theta[4]) - yd;\n",
    "Df(theta) = hcat(\n",
    "    exp.(theta[2]*xd) .* cos.(theta[3] * xd .+ theta[4]),\n",
    "    theta[1] * ( xd .* exp.(theta[2]*xd) .*\n",
    "        cos.(theta[3] * xd .+ theta[4])),\n",
    "    -theta[1] * ( exp.(theta[2]*xd) .* xd .*\n",
    "        sin.(theta[3] * xd .+ theta[4])),\n",
    "    -theta[1] * ( exp.(theta[2]*xd) .*\n",
    "        sin.(theta[3] * xd .+ theta[4])) );\n",
    "theta1 = [1, 0, 1, 0];\n",
    "theta, history = levenberg_marquardt(f, Df, theta1, 1.0)\n",
    "theta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"600\" height=\"400\" viewBox=\"0 0 2400 1600\">\n",
       "<defs>\n",
       "  <clipPath id=\"clip6300\">\n",
       "    <rect x=\"0\" y=\"0\" width=\"2400\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<path clip-path=\"url(#clip6300)\" d=\"\n",
       "M0 1600 L2400 1600 L2400 0 L0 0  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip6301\">\n",
       "    <rect x=\"480\" y=\"0\" width=\"1681\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<path clip-path=\"url(#clip6300)\" d=\"\n",
       "M182.008 1487.47 L2352.76 1487.47 L2352.76 47.2441 L182.008 47.2441  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip6302\">\n",
       "    <rect x=\"182\" y=\"47\" width=\"2172\" height=\"1441\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polyline clip-path=\"url(#clip6302)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  243.444,1487.47 243.444,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6302)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  755.413,1487.47 755.413,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6302)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1267.38,1487.47 1267.38,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6302)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1779.35,1487.47 1779.35,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6302)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  2291.32,1487.47 2291.32,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6302)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  182.008,1227.32 2352.76,1227.32 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6302)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  182.008,944.153 2352.76,944.153 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6302)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  182.008,660.989 2352.76,660.989 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6302)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  182.008,377.826 2352.76,377.826 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6302)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  182.008,94.6629 2352.76,94.6629 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6300)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  182.008,1487.47 2352.76,1487.47 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6300)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  182.008,1487.47 182.008,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6300)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  243.444,1487.47 243.444,1465.87 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6300)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  755.413,1487.47 755.413,1465.87 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6300)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1267.38,1487.47 1267.38,1465.87 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6300)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1779.35,1487.47 1779.35,1465.87 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6300)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  2291.32,1487.47 2291.32,1465.87 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6300)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  182.008,1227.32 214.569,1227.32 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6300)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  182.008,944.153 214.569,944.153 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6300)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  182.008,660.989 214.569,660.989 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6300)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  182.008,377.826 214.569,377.826 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6300)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  182.008,94.6629 214.569,94.6629 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip6300)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 243.444, 1541.47)\" x=\"243.444\" y=\"1541.47\">0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip6300)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 755.413, 1541.47)\" x=\"755.413\" y=\"1541.47\">5</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip6300)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 1267.38, 1541.47)\" x=\"1267.38\" y=\"1541.47\">10</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip6300)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 1779.35, 1541.47)\" x=\"1779.35\" y=\"1541.47\">15</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip6300)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 2291.32, 1541.47)\" x=\"2291.32\" y=\"1541.47\">20</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip6300)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 158.008, 1244.82)\" x=\"158.008\" y=\"1244.82\">-0.6</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip6300)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 158.008, 961.653)\" x=\"158.008\" y=\"961.653\">-0.3</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip6300)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 158.008, 678.489)\" x=\"158.008\" y=\"678.489\">0.0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip6300)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 158.008, 395.326)\" x=\"158.008\" y=\"395.326\">0.3</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip6300)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 158.008, 112.163)\" x=\"158.008\" y=\"112.163\">0.6</text>\n",
       "</g>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"427.754\" cy=\"1287.85\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"427.754\" cy=\"1287.85\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"327.214\" cy=\"1109.54\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"327.214\" cy=\"1109.54\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"285.922\" cy=\"681.087\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"285.922\" cy=\"681.087\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"551.709\" cy=\"569.475\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"551.709\" cy=\"569.475\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"489.565\" cy=\"947.296\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"489.565\" cy=\"947.296\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"253.858\" cy=\"336.899\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"253.858\" cy=\"336.899\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"404.355\" cy=\"1446.71\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"404.355\" cy=\"1446.71\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"721.786\" cy=\"293.39\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"721.786\" cy=\"293.39\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"661.299\" cy=\"88.0053\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"661.299\" cy=\"88.0053\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"294.971\" cy=\"694.306\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"294.971\" cy=\"694.306\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"283.878\" cy=\"634.987\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"283.878\" cy=\"634.987\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"249.731\" cy=\"362.726\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"249.731\" cy=\"362.726\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"660.413\" cy=\"102.018\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"660.413\" cy=\"102.018\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"387.31\" cy=\"1317.89\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"387.31\" cy=\"1317.89\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"422.36\" cy=\"1246.81\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"422.36\" cy=\"1246.81\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"418.505\" cy=\"1327.92\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"418.505\" cy=\"1327.92\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"649.914\" cy=\"302.03\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"649.914\" cy=\"302.03\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"400.082\" cy=\"1176.08\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"400.082\" cy=\"1176.08\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"596.8\" cy=\"412.295\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"596.8\" cy=\"412.295\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"570.559\" cy=\"534.183\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"570.559\" cy=\"534.183\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"579.034\" cy=\"506.347\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"579.034\" cy=\"506.347\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"729.995\" cy=\"311.834\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"729.995\" cy=\"311.834\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"462.63\" cy=\"1150.91\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"462.63\" cy=\"1150.91\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"607.439\" cy=\"480.765\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"607.439\" cy=\"480.765\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"281.279\" cy=\"613.168\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"281.279\" cy=\"613.168\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"552.601\" cy=\"607.479\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"552.601\" cy=\"607.479\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"355.522\" cy=\"1261.31\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"355.522\" cy=\"1261.31\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"346.22\" cy=\"1193.01\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"346.22\" cy=\"1193.01\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"455.315\" cy=\"1120.69\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"455.315\" cy=\"1120.69\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"526.041\" cy=\"788.941\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"526.041\" cy=\"788.941\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1880.98\" cy=\"704.636\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1880.98\" cy=\"704.636\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1169.75\" cy=\"472.336\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1169.75\" cy=\"472.336\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1092.23\" cy=\"562.752\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1092.23\" cy=\"562.752\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1503.06\" cy=\"696.464\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1503.06\" cy=\"696.464\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2140.34\" cy=\"637.882\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2140.34\" cy=\"637.882\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1367.15\" cy=\"745.854\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1367.15\" cy=\"745.854\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"963.025\" cy=\"889.588\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"963.025\" cy=\"889.588\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2112.16\" cy=\"661.087\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2112.16\" cy=\"661.087\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1999.81\" cy=\"681.882\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1999.81\" cy=\"681.882\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1891.83\" cy=\"678.432\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1891.83\" cy=\"678.432\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1792.5\" cy=\"653.29\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1792.5\" cy=\"653.29\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1050.13\" cy=\"688.688\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1050.13\" cy=\"688.688\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"969.962\" cy=\"854.647\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"969.962\" cy=\"854.647\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1430.41\" cy=\"747.768\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1430.41\" cy=\"747.768\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2228.52\" cy=\"653.642\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2228.52\" cy=\"653.642\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"910.586\" cy=\"887.633\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"910.586\" cy=\"887.633\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"825.675\" cy=\"772.347\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"825.675\" cy=\"772.347\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"910.754\" cy=\"946.46\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"910.754\" cy=\"946.46\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1446.26\" cy=\"744.457\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1446.26\" cy=\"744.457\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1896.99\" cy=\"692.139\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1896.99\" cy=\"692.139\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1682.25\" cy=\"611.962\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1682.25\" cy=\"611.962\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1469.51\" cy=\"779.182\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1469.51\" cy=\"779.182\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1306.48\" cy=\"651.954\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1306.48\" cy=\"651.954\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"789.689\" cy=\"647.027\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"789.689\" cy=\"647.027\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1176.21\" cy=\"524.729\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1176.21\" cy=\"524.729\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1446.17\" cy=\"757.868\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1446.17\" cy=\"757.868\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1034.12\" cy=\"714.25\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1034.12\" cy=\"714.25\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1041.05\" cy=\"721.34\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1041.05\" cy=\"721.34\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1027.96\" cy=\"733.997\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1027.96\" cy=\"733.997\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1003.5\" cy=\"796.46\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6302)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1003.5\" cy=\"796.46\" r=\"14\"/>\n",
       "<polyline clip-path=\"url(#clip6302)\" style=\"stroke:#e26f46; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  243.444,243.236 247.548,285.049 251.652,327.208 255.756,369.604 259.86,412.126 263.964,454.664 268.068,497.113 272.172,539.366 276.276,581.319 280.38,622.873 \n",
       "  284.484,663.927 288.588,704.386 292.692,744.156 296.796,783.147 300.9,821.273 305.004,858.45 309.108,894.597 313.212,929.639 317.316,963.503 321.42,996.122 \n",
       "  325.523,1027.43 329.627,1057.37 333.731,1085.88 337.835,1112.91 341.939,1138.43 346.043,1162.37 350.147,1184.71 354.251,1205.41 358.355,1224.45 362.459,1241.8 \n",
       "  366.563,1257.43 370.667,1271.35 374.771,1283.52 378.875,1293.96 382.979,1302.66 387.083,1309.61 391.187,1314.84 395.291,1318.34 399.395,1320.15 403.499,1320.26 \n",
       "  407.603,1318.72 411.707,1315.55 415.811,1310.77 419.915,1304.43 424.018,1296.56 428.122,1287.2 432.226,1276.4 436.33,1264.21 440.434,1250.68 444.538,1235.86 \n",
       "  448.642,1219.8 452.746,1202.56 456.85,1184.21 460.954,1164.81 465.058,1144.42 469.162,1123.1 473.266,1100.93 477.37,1077.97 481.474,1054.28 485.578,1029.95 \n",
       "  489.682,1005.04 493.786,979.616 497.89,953.756 501.994,927.531 506.098,901.011 510.202,874.266 514.306,847.368 518.41,820.387 522.514,793.391 526.617,766.448 \n",
       "  530.721,739.626 534.825,712.99 538.929,686.605 543.033,660.533 547.137,634.836 551.241,609.573 555.345,584.8 559.449,560.574 563.553,536.948 567.657,513.972 \n",
       "  571.761,491.695 575.865,470.163 579.969,449.419 584.073,429.505 588.177,410.458 592.281,392.314 596.385,375.106 600.489,358.864 604.593,343.614 608.697,329.381 \n",
       "  612.801,316.186 616.905,304.048 621.009,292.981 625.112,282.998 629.216,274.11 633.32,266.321 637.424,259.638 641.528,254.059 645.632,249.585 649.736,246.209 \n",
       "  653.84,243.925 657.944,242.723 662.048,242.591 666.152,243.513 670.256,245.472 674.36,248.448 678.464,252.419 682.568,257.362 686.672,263.249 690.776,270.052 \n",
       "  694.88,277.741 698.984,286.284 703.088,295.648 707.192,305.797 711.296,316.693 715.4,328.3 719.504,340.578 723.607,353.486 727.711,366.982 731.815,381.025 \n",
       "  735.919,395.571 740.023,410.577 744.127,425.997 748.231,441.788 752.335,457.905 756.439,474.301 760.543,490.932 764.647,507.753 768.751,524.718 772.855,541.783 \n",
       "  776.959,558.904 781.063,576.036 785.167,593.137 789.271,610.164 793.375,627.074 797.479,643.828 801.583,660.385 805.687,676.706 809.791,692.754 813.895,708.493 \n",
       "  817.999,723.886 822.102,738.901 826.206,753.505 830.31,767.666 834.414,781.357 838.518,794.549 842.622,807.216 846.726,819.333 850.83,830.879 854.934,841.832 \n",
       "  859.038,852.174 863.142,861.886 867.246,870.953 871.35,879.363 875.454,887.103 879.558,894.163 883.662,900.536 887.766,906.214 891.87,911.195 895.974,915.474 \n",
       "  900.078,919.052 904.182,921.93 908.286,924.11 912.39,925.597 916.494,926.397 920.597,926.518 924.701,925.969 928.805,924.761 932.909,922.907 937.013,920.421 \n",
       "  941.117,917.318 945.221,913.614 949.325,909.328 953.429,904.478 957.533,899.085 961.637,893.171 965.741,886.757 969.845,879.867 973.949,872.525 978.053,864.757 \n",
       "  982.157,856.586 986.261,848.041 990.365,839.148 994.469,829.933 998.573,820.426 1002.68,810.654 1006.78,800.645 1010.88,790.428 1014.99,780.032 1019.09,769.485 \n",
       "  1023.2,758.817 1027.3,748.055 1031.4,737.228 1035.51,726.364 1039.61,715.492 1043.72,704.638 1047.82,693.829 1051.92,683.093 1056.03,672.455 1060.13,661.941 \n",
       "  1064.24,651.575 1068.34,641.381 1072.44,631.382 1076.55,621.601 1080.65,612.059 1084.76,602.776 1088.86,593.774 1092.96,585.069 1097.07,576.68 1101.17,568.623 \n",
       "  1105.28,560.913 1109.38,553.566 1113.48,546.595 1117.59,540.011 1121.69,533.826 1125.8,528.049 1129.9,522.689 1134,517.754 1138.11,513.25 1142.21,509.182 \n",
       "  1146.32,505.555 1150.42,502.37 1154.52,499.63 1158.63,497.335 1162.73,495.485 1166.83,494.077 1170.94,493.11 1175.04,492.578 1179.15,492.479 1183.25,492.804 \n",
       "  1187.35,493.548 1191.46,494.702 1195.56,496.258 1199.67,498.207 1203.77,500.537 1207.87,503.237 1211.98,506.295 1216.08,509.699 1220.19,513.435 1224.29,517.488 \n",
       "  1228.39,521.844 1232.5,526.488 1236.6,531.404 1240.71,536.575 1244.81,541.986 1248.91,547.618 1253.02,553.455 1257.12,559.478 1261.23,565.671 1265.33,572.015 \n",
       "  1269.43,578.492 1273.54,585.084 1277.64,591.772 1281.75,598.538 1285.85,605.365 1289.95,612.234 1294.06,619.127 1298.16,626.027 1302.27,632.916 1306.37,639.777 \n",
       "  1310.47,646.593 1314.58,653.348 1318.68,660.025 1322.79,666.609 1326.89,673.084 1330.99,679.437 1335.1,685.652 1339.2,691.716 1343.31,697.615 1347.41,703.339 \n",
       "  1351.51,708.873 1355.62,714.208 1359.72,719.333 1363.83,724.237 1367.93,728.912 1372.03,733.35 1376.14,737.542 1380.24,741.481 1384.34,745.161 1388.45,748.577 \n",
       "  1392.55,751.723 1396.66,754.596 1400.76,757.193 1404.86,759.51 1408.97,761.547 1413.07,763.301 1417.18,764.773 1421.28,765.962 1425.38,766.87 1429.49,767.5 \n",
       "  1433.59,767.852 1437.7,767.93 1441.8,767.738 1445.9,767.28 1450.01,766.562 1454.11,765.588 1458.22,764.365 1462.32,762.899 1466.42,761.198 1470.53,759.269 \n",
       "  1474.63,757.121 1478.74,754.762 1482.84,752.2 1486.94,749.446 1491.05,746.509 1495.15,743.398 1499.26,740.125 1503.36,736.7 1507.46,733.133 1511.57,729.435 \n",
       "  1515.67,725.619 1519.78,721.695 1523.88,717.674 1527.98,713.568 1532.09,709.388 1536.19,705.147 1540.3,700.855 1544.4,696.525 1548.5,692.167 1552.61,687.793 \n",
       "  1556.71,683.414 1560.82,679.042 1564.92,674.687 1569.02,670.36 1573.13,666.071 1577.23,661.831 1581.33,657.649 1585.44,653.536 1589.54,649.5 1593.65,645.551 \n",
       "  1597.75,641.697 1601.85,637.947 1605.96,634.309 1610.06,630.79 1614.17,627.398 1618.27,624.138 1622.37,621.018 1626.48,618.043 1630.58,615.219 1634.69,612.55 \n",
       "  1638.79,610.042 1642.89,607.697 1647,605.52 1651.1,603.514 1655.21,601.681 1659.31,600.024 1663.41,598.544 1667.52,597.242 1671.62,596.119 1675.73,595.175 \n",
       "  1679.83,594.41 1683.93,593.824 1688.04,593.415 1692.14,593.182 1696.25,593.123 1700.35,593.236 1704.45,593.517 1708.56,593.964 1712.66,594.574 1716.77,595.341 \n",
       "  1720.87,596.263 1724.97,597.335 1729.08,598.551 1733.18,599.907 1737.29,601.397 1741.39,603.015 1745.49,604.757 1749.6,606.615 1753.7,608.583 1757.81,610.655 \n",
       "  1761.91,612.823 1766.01,615.082 1770.12,617.424 1774.22,619.842 1778.32,622.329 1782.43,624.877 1786.53,627.48 1790.64,630.13 1794.74,632.82 1798.84,635.542 \n",
       "  1802.95,638.289 1807.05,641.053 1811.16,643.829 1815.26,646.607 1819.36,649.383 1823.47,652.147 1827.57,654.894 1831.68,657.618 1835.78,660.31 1839.88,662.966 \n",
       "  1843.99,665.579 1848.09,668.143 1852.2,670.652 1856.3,673.101 1860.4,675.484 1864.51,677.797 1868.61,680.034 1872.72,682.192 1876.82,684.265 1880.92,686.25 \n",
       "  1885.03,688.143 1889.13,689.94 1893.24,691.639 1897.34,693.237 1901.44,694.731 1905.55,696.118 1909.65,697.397 1913.76,698.566 1917.86,699.624 1921.96,700.569 \n",
       "  1926.07,701.402 1930.17,702.121 1934.28,702.726 1938.38,703.217 1942.48,703.595 1946.59,703.861 1950.69,704.015 1954.8,704.058 1958.9,703.992 1963,703.82 \n",
       "  1967.11,703.542 1971.21,703.16 1975.31,702.679 1979.42,702.099 1983.52,701.424 1987.63,700.657 1991.73,699.802 1995.83,698.86 1999.94,697.838 2004.04,696.737 \n",
       "  2008.15,695.562 2012.25,694.316 2016.35,693.005 2020.46,691.632 2024.56,690.202 2028.67,688.718 2032.77,687.186 2036.87,685.61 2040.98,683.995 2045.08,682.345 \n",
       "  2049.19,680.665 2053.29,678.959 2057.39,677.233 2061.5,675.49 2065.6,673.736 2069.71,671.975 2073.81,670.212 2077.91,668.451 2082.02,666.696 2086.12,664.952 \n",
       "  2090.23,663.223 2094.33,661.513 2098.43,659.826 2102.54,658.166 2106.64,656.538 2110.75,654.943 2114.85,653.387 2118.95,651.872 2123.06,650.402 2127.16,648.979 \n",
       "  2131.27,647.608 2135.37,646.289 2139.47,645.026 2143.58,643.822 2147.68,642.678 2151.79,641.596 2155.89,640.579 2159.99,639.627 2164.1,638.743 2168.2,637.928 \n",
       "  2172.3,637.182 2176.41,636.507 2180.51,635.903 2184.62,635.371 2188.72,634.911 2192.82,634.523 2196.93,634.207 2201.03,633.963 2205.14,633.791 2209.24,633.689 \n",
       "  2213.34,633.658 2217.45,633.696 2221.55,633.802 2225.66,633.975 2229.76,634.213 2233.86,634.515 2237.97,634.88 2242.07,635.305 2246.18,635.788 2250.28,636.328 \n",
       "  2254.38,636.923 2258.49,637.569 2262.59,638.265 2266.7,639.008 2270.8,639.796 2274.9,640.626 2279.01,641.495 2283.11,642.401 2287.22,643.341 2291.32,644.311 \n",
       "  \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6302)\" style=\"stroke:#3da44d; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  243.444,243.236 247.548,285.049 251.652,327.208 255.756,369.604 259.86,412.126 263.964,454.664 268.068,497.113 272.172,539.366 276.276,581.319 280.38,622.873 \n",
       "  284.484,663.927 288.588,704.386 292.692,744.156 296.796,783.147 300.9,821.273 305.004,858.45 309.108,894.597 313.212,929.639 317.316,963.503 321.42,996.122 \n",
       "  325.523,1027.43 329.627,1057.37 333.731,1085.88 337.835,1112.91 341.939,1138.43 346.043,1162.37 350.147,1184.71 354.251,1205.41 358.355,1224.45 362.459,1241.8 \n",
       "  366.563,1257.43 370.667,1271.35 374.771,1283.52 378.875,1293.96 382.979,1302.66 387.083,1309.61 391.187,1314.84 395.291,1318.34 399.395,1320.15 403.499,1320.26 \n",
       "  407.603,1318.72 411.707,1315.55 415.811,1310.77 419.915,1304.43 424.018,1296.56 428.122,1287.2 432.226,1276.4 436.33,1264.21 440.434,1250.68 444.538,1235.86 \n",
       "  448.642,1219.8 452.746,1202.56 456.85,1184.21 460.954,1164.81 465.058,1144.42 469.162,1123.1 473.266,1100.93 477.37,1077.97 481.474,1054.28 485.578,1029.95 \n",
       "  489.682,1005.04 493.786,979.616 497.89,953.756 501.994,927.531 506.098,901.011 510.202,874.266 514.306,847.368 518.41,820.387 522.514,793.391 526.617,766.448 \n",
       "  530.721,739.626 534.825,712.99 538.929,686.605 543.033,660.533 547.137,634.836 551.241,609.573 555.345,584.8 559.449,560.574 563.553,536.948 567.657,513.972 \n",
       "  571.761,491.695 575.865,470.163 579.969,449.419 584.073,429.505 588.177,410.458 592.281,392.314 596.385,375.106 600.489,358.864 604.593,343.614 608.697,329.381 \n",
       "  612.801,316.186 616.905,304.048 621.009,292.981 625.112,282.998 629.216,274.11 633.32,266.321 637.424,259.638 641.528,254.059 645.632,249.585 649.736,246.209 \n",
       "  653.84,243.925 657.944,242.723 662.048,242.591 666.152,243.513 670.256,245.472 674.36,248.448 678.464,252.419 682.568,257.362 686.672,263.249 690.776,270.052 \n",
       "  694.88,277.741 698.984,286.284 703.088,295.648 707.192,305.797 711.296,316.693 715.4,328.3 719.504,340.578 723.607,353.486 727.711,366.982 731.815,381.025 \n",
       "  735.919,395.571 740.023,410.577 744.127,425.997 748.231,441.788 752.335,457.905 756.439,474.301 760.543,490.932 764.647,507.753 768.751,524.718 772.855,541.783 \n",
       "  776.959,558.904 781.063,576.036 785.167,593.137 789.271,610.164 793.375,627.074 797.479,643.828 801.583,660.385 805.687,676.706 809.791,692.754 813.895,708.493 \n",
       "  817.999,723.886 822.102,738.901 826.206,753.505 830.31,767.666 834.414,781.357 838.518,794.549 842.622,807.216 846.726,819.333 850.83,830.879 854.934,841.832 \n",
       "  859.038,852.174 863.142,861.886 867.246,870.953 871.35,879.363 875.454,887.103 879.558,894.163 883.662,900.536 887.766,906.214 891.87,911.195 895.974,915.474 \n",
       "  900.078,919.052 904.182,921.93 908.286,924.11 912.39,925.597 916.494,926.397 920.597,926.518 924.701,925.969 928.805,924.761 932.909,922.907 937.013,920.421 \n",
       "  941.117,917.318 945.221,913.614 949.325,909.328 953.429,904.478 957.533,899.085 961.637,893.171 965.741,886.757 969.845,879.867 973.949,872.525 978.053,864.757 \n",
       "  982.157,856.586 986.261,848.041 990.365,839.148 994.469,829.933 998.573,820.426 1002.68,810.654 1006.78,800.645 1010.88,790.428 1014.99,780.032 1019.09,769.485 \n",
       "  1023.2,758.817 1027.3,748.055 1031.4,737.228 1035.51,726.364 1039.61,715.492 1043.72,704.638 1047.82,693.829 1051.92,683.093 1056.03,672.455 1060.13,661.941 \n",
       "  1064.24,651.575 1068.34,641.381 1072.44,631.382 1076.55,621.601 1080.65,612.059 1084.76,602.776 1088.86,593.774 1092.96,585.069 1097.07,576.68 1101.17,568.623 \n",
       "  1105.28,560.913 1109.38,553.566 1113.48,546.595 1117.59,540.011 1121.69,533.826 1125.8,528.049 1129.9,522.689 1134,517.754 1138.11,513.25 1142.21,509.182 \n",
       "  1146.32,505.555 1150.42,502.37 1154.52,499.63 1158.63,497.335 1162.73,495.485 1166.83,494.077 1170.94,493.11 1175.04,492.578 1179.15,492.479 1183.25,492.804 \n",
       "  1187.35,493.548 1191.46,494.702 1195.56,496.258 1199.67,498.207 1203.77,500.537 1207.87,503.237 1211.98,506.295 1216.08,509.699 1220.19,513.435 1224.29,517.488 \n",
       "  1228.39,521.844 1232.5,526.488 1236.6,531.404 1240.71,536.575 1244.81,541.986 1248.91,547.618 1253.02,553.455 1257.12,559.478 1261.23,565.671 1265.33,572.015 \n",
       "  1269.43,578.492 1273.54,585.084 1277.64,591.772 1281.75,598.538 1285.85,605.365 1289.95,612.234 1294.06,619.127 1298.16,626.027 1302.27,632.916 1306.37,639.777 \n",
       "  1310.47,646.593 1314.58,653.348 1318.68,660.025 1322.79,666.609 1326.89,673.084 1330.99,679.437 1335.1,685.652 1339.2,691.716 1343.31,697.615 1347.41,703.339 \n",
       "  1351.51,708.873 1355.62,714.208 1359.72,719.333 1363.83,724.237 1367.93,728.912 1372.03,733.35 1376.14,737.542 1380.24,741.481 1384.34,745.161 1388.45,748.577 \n",
       "  1392.55,751.723 1396.66,754.596 1400.76,757.193 1404.86,759.51 1408.97,761.547 1413.07,763.301 1417.18,764.773 1421.28,765.962 1425.38,766.87 1429.49,767.5 \n",
       "  1433.59,767.852 1437.7,767.93 1441.8,767.738 1445.9,767.28 1450.01,766.562 1454.11,765.588 1458.22,764.365 1462.32,762.899 1466.42,761.198 1470.53,759.269 \n",
       "  1474.63,757.121 1478.74,754.762 1482.84,752.2 1486.94,749.446 1491.05,746.509 1495.15,743.398 1499.26,740.125 1503.36,736.7 1507.46,733.133 1511.57,729.435 \n",
       "  1515.67,725.619 1519.78,721.695 1523.88,717.674 1527.98,713.568 1532.09,709.388 1536.19,705.147 1540.3,700.855 1544.4,696.525 1548.5,692.167 1552.61,687.793 \n",
       "  1556.71,683.414 1560.82,679.042 1564.92,674.687 1569.02,670.36 1573.13,666.071 1577.23,661.831 1581.33,657.649 1585.44,653.536 1589.54,649.5 1593.65,645.551 \n",
       "  1597.75,641.697 1601.85,637.947 1605.96,634.309 1610.06,630.79 1614.17,627.398 1618.27,624.138 1622.37,621.018 1626.48,618.043 1630.58,615.219 1634.69,612.55 \n",
       "  1638.79,610.042 1642.89,607.697 1647,605.52 1651.1,603.514 1655.21,601.681 1659.31,600.024 1663.41,598.544 1667.52,597.242 1671.62,596.119 1675.73,595.175 \n",
       "  1679.83,594.41 1683.93,593.824 1688.04,593.415 1692.14,593.182 1696.25,593.123 1700.35,593.236 1704.45,593.517 1708.56,593.964 1712.66,594.574 1716.77,595.341 \n",
       "  1720.87,596.263 1724.97,597.335 1729.08,598.551 1733.18,599.907 1737.29,601.397 1741.39,603.015 1745.49,604.757 1749.6,606.615 1753.7,608.583 1757.81,610.655 \n",
       "  1761.91,612.823 1766.01,615.082 1770.12,617.424 1774.22,619.842 1778.32,622.329 1782.43,624.877 1786.53,627.48 1790.64,630.13 1794.74,632.82 1798.84,635.542 \n",
       "  1802.95,638.289 1807.05,641.053 1811.16,643.829 1815.26,646.607 1819.36,649.383 1823.47,652.147 1827.57,654.894 1831.68,657.618 1835.78,660.31 1839.88,662.966 \n",
       "  1843.99,665.579 1848.09,668.143 1852.2,670.652 1856.3,673.101 1860.4,675.484 1864.51,677.797 1868.61,680.034 1872.72,682.192 1876.82,684.265 1880.92,686.25 \n",
       "  1885.03,688.143 1889.13,689.94 1893.24,691.639 1897.34,693.237 1901.44,694.731 1905.55,696.118 1909.65,697.397 1913.76,698.566 1917.86,699.624 1921.96,700.569 \n",
       "  1926.07,701.402 1930.17,702.121 1934.28,702.726 1938.38,703.217 1942.48,703.595 1946.59,703.861 1950.69,704.015 1954.8,704.058 1958.9,703.992 1963,703.82 \n",
       "  1967.11,703.542 1971.21,703.16 1975.31,702.679 1979.42,702.099 1983.52,701.424 1987.63,700.657 1991.73,699.802 1995.83,698.86 1999.94,697.838 2004.04,696.737 \n",
       "  2008.15,695.562 2012.25,694.316 2016.35,693.005 2020.46,691.632 2024.56,690.202 2028.67,688.718 2032.77,687.186 2036.87,685.61 2040.98,683.995 2045.08,682.345 \n",
       "  2049.19,680.665 2053.29,678.959 2057.39,677.233 2061.5,675.49 2065.6,673.736 2069.71,671.975 2073.81,670.212 2077.91,668.451 2082.02,666.696 2086.12,664.952 \n",
       "  2090.23,663.223 2094.33,661.513 2098.43,659.826 2102.54,658.166 2106.64,656.538 2110.75,654.943 2114.85,653.387 2118.95,651.872 2123.06,650.402 2127.16,648.979 \n",
       "  2131.27,647.608 2135.37,646.289 2139.47,645.026 2143.58,643.822 2147.68,642.678 2151.79,641.596 2155.89,640.579 2159.99,639.627 2164.1,638.743 2168.2,637.928 \n",
       "  2172.3,637.182 2176.41,636.507 2180.51,635.903 2184.62,635.371 2188.72,634.911 2192.82,634.523 2196.93,634.207 2201.03,633.963 2205.14,633.791 2209.24,633.689 \n",
       "  2213.34,633.658 2217.45,633.696 2221.55,633.802 2225.66,633.975 2229.76,634.213 2233.86,634.515 2237.97,634.88 2242.07,635.305 2246.18,635.788 2250.28,636.328 \n",
       "  2254.38,636.923 2258.49,637.569 2262.59,638.265 2266.7,639.008 2270.8,639.796 2274.9,640.626 2279.01,641.495 2283.11,642.401 2287.22,643.341 2291.32,644.311 \n",
       "  \n",
       "  \"/>\n",
       "</svg>\n"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Plot the fitted model.\n",
    "x = linspace(0, 20, 500);\n",
    "y=theta[1]*exp.(theta[2]*x) .* cos.(theta[3]*x .+ theta[4]);\n",
    "plot!(x, y, legend = false)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Figure 18.<a ind=\"#fig\"></a>5** Least squares fit of a function $f̂(x; θ) = θ_1e^{θ_{2x}}cos(θ_3x+ θ_4)$ to\n",
    "$N = 60$ points $(x^{(i)}, y^{(i)})$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Orthogonal distance regression.** In figure [18.14](https://web.stanford.edu/~boyd/vmls/vmls.pdf#figure.18.14) of VMLS we use orthogonal distance regression to fit a cubic polynomial\n",
    "\n",
    "$$\n",
    "f̂(x; θ) = θ_1 + θ_2x+ θ_3x^2 + θ_4x^3\n",
    "$$\n",
    "\n",
    "to $N = 25$ data points.\n",
    "We first read in the data and compute the standard least squares fit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "xd, yd = orth_dist_reg_data(); # 2 vectors of length N = 25\n",
    "N = length(xd);\n",
    "p = 4;\n",
    "theta_ls = vandermonde(xd, p) \\ yd;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The nonlinear least squares formulation on page 400 of VMLS has $p + N$ variables $\\theta_1,...,\\theta_p, u^{(1)}, ..., u^{(N)}$. We will store them in that order in the nonlinear least squares vector variable. The objective is to minimize the squared norm of the $2N$-vector\n",
    "\n",
    "\\begin{bmatrix}\n",
    "   \\hat{f}(u^{(1)}; \\theta) - y^{(1)} \\\\\n",
    "    \\vdots \\\\\n",
    "    \\hat{f}(u^{(N)}; \\theta) - y^{(N)} \\\\\n",
    "    u^{(1)} - x^{(1)} \\\\\n",
    "    \\vdots \\\\\n",
    "    u^{(N)} - x^{(N)}\n",
    "\\end{bmatrix}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "function f(x)\n",
    "    theta = x[1:p];\n",
    "    u = x[p+1:end];\n",
    "    f1 = vandermonde(u,p)*theta - yd\n",
    "    f2 = u - xd\n",
    "    return [f1; f2]\n",
    "    end;\n",
    "\n",
    "function Df(x)\n",
    "    theta = x[1:p]\n",
    "    u = x[p+1:end]\n",
    "    D11 = vandermonde(u,p)\n",
    "    D12 = diagonal(theta[2] .+ 2*theta[3]*u .+ 3*theta[4]*u.^2)\n",
    "    D21 = zeros(N,p)\n",
    "    D22 = eye(N)\n",
    "    return [ D11 D12; D21 D22]\n",
    "    end;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now call `levenberg-marquardt` with these two functions. A natural choice for the initial point is to use the least squares solution for the variables $\\theta$ and the data points $x^{(i)}$ for the variables $u^{(i)}$. We use $\\lambda^{(1)} = 0.01$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "sol, hist = levenberg_marquardt(f, Df, [theta_ls; xd], 0.01);\n",
    "theta_od = sol[1:p];"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Figure [18.6](#fig6) shows the two fitted polynomials."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"600\" height=\"400\" viewBox=\"0 0 2400 1600\">\n",
       "<defs>\n",
       "  <clipPath id=\"clip6700\">\n",
       "    <rect x=\"0\" y=\"0\" width=\"2400\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<path clip-path=\"url(#clip6700)\" d=\"\n",
       "M0 1600 L2400 1600 L2400 0 L0 0  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip6701\">\n",
       "    <rect x=\"480\" y=\"0\" width=\"1681\" height=\"1600\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<path clip-path=\"url(#clip6700)\" d=\"\n",
       "M182.008 1487.47 L2352.76 1487.47 L2352.76 47.2441 L182.008 47.2441  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<defs>\n",
       "  <clipPath id=\"clip6702\">\n",
       "    <rect x=\"182\" y=\"47\" width=\"2172\" height=\"1441\"/>\n",
       "  </clipPath>\n",
       "</defs>\n",
       "<polyline clip-path=\"url(#clip6702)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  423.875,1487.47 423.875,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6702)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  900.407,1487.47 900.407,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6702)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1376.94,1487.47 1376.94,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6702)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  1853.47,1487.47 1853.47,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6702)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  2330,1487.47 2330,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6702)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  182.008,1402.97 2352.76,1402.97 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6702)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  182.008,1128.7 2352.76,1128.7 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6702)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  182.008,854.443 2352.76,854.443 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6702)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  182.008,580.181 2352.76,580.181 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6702)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
       "  182.008,305.92 2352.76,305.92 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6700)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  182.008,1487.47 2352.76,1487.47 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6700)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  182.008,1487.47 182.008,47.2441 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6700)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  423.875,1487.47 423.875,1465.87 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6700)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  900.407,1487.47 900.407,1465.87 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6700)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1376.94,1487.47 1376.94,1465.87 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6700)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  1853.47,1487.47 1853.47,1465.87 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6700)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  2330,1487.47 2330,1465.87 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6700)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  182.008,1402.97 214.569,1402.97 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6700)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  182.008,1128.7 214.569,1128.7 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6700)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  182.008,854.443 214.569,854.443 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6700)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  182.008,580.181 214.569,580.181 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6700)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  182.008,305.92 214.569,305.92 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip6700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 423.875, 1541.47)\" x=\"423.875\" y=\"1541.47\">0.2</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip6700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 900.407, 1541.47)\" x=\"900.407\" y=\"1541.47\">0.4</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip6700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 1376.94, 1541.47)\" x=\"1376.94\" y=\"1541.47\">0.6</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip6700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 1853.47, 1541.47)\" x=\"1853.47\" y=\"1541.47\">0.8</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip6700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:middle;\" transform=\"rotate(0, 2330, 1541.47)\" x=\"2330\" y=\"1541.47\">1.0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip6700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 158.008, 1420.47)\" x=\"158.008\" y=\"1420.47\">-0.2</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip6700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 158.008, 1146.2)\" x=\"158.008\" y=\"1146.2\">-0.1</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip6700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 158.008, 871.943)\" x=\"158.008\" y=\"871.943\">0.0</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip6700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 158.008, 597.681)\" x=\"158.008\" y=\"597.681\">0.1</text>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip6700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:end;\" transform=\"rotate(0, 158.008, 323.42)\" x=\"158.008\" y=\"323.42\">0.2</text>\n",
       "</g>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2005.64\" cy=\"97.1658\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2005.64\" cy=\"97.1658\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2106.48\" cy=\"453.806\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2106.48\" cy=\"453.806\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"424.228\" cy=\"1327.54\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"424.228\" cy=\"1327.54\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1994.99\" cy=\"508.649\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1994.99\" cy=\"508.649\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1843.83\" cy=\"275.311\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1843.83\" cy=\"275.311\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"753.153\" cy=\"1338.5\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"753.153\" cy=\"1338.5\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1685.07\" cy=\"204.39\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1685.07\" cy=\"204.39\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1240.53\" cy=\"546.398\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1240.53\" cy=\"546.398\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"972.365\" cy=\"1378.77\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"972.365\" cy=\"1378.77\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"891.682\" cy=\"1232.56\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"891.682\" cy=\"1232.56\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"456.627\" cy=\"1095.7\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"456.627\" cy=\"1095.7\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"266.042\" cy=\"1239.88\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"266.042\" cy=\"1239.88\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2050.57\" cy=\"247.194\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2050.57\" cy=\"247.194\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1888.61\" cy=\"88.3239\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1888.61\" cy=\"88.3239\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1723.25\" cy=\"134.501\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1723.25\" cy=\"134.501\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2272.93\" cy=\"1037.78\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2272.93\" cy=\"1037.78\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"243.444\" cy=\"951.81\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"243.444\" cy=\"951.81\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1451.41\" cy=\"282.989\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1451.41\" cy=\"282.989\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2165.59\" cy=\"701.633\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2165.59\" cy=\"701.633\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2009.58\" cy=\"569.717\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2009.58\" cy=\"569.717\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1141.61\" cy=\"717.678\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1141.61\" cy=\"717.678\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1149.86\" cy=\"726.616\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1149.86\" cy=\"726.616\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"2291.32\" cy=\"835.761\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"2291.32\" cy=\"835.761\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1954.24\" cy=\"146.847\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1954.24\" cy=\"146.847\" r=\"14\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#000000; stroke:none; fill-opacity:1\" cx=\"1022.38\" cy=\"1115.98\" r=\"18\"/>\n",
       "<circle clip-path=\"url(#clip6702)\" style=\"fill:#009af9; stroke:none; fill-opacity:1\" cx=\"1022.38\" cy=\"1115.98\" r=\"14\"/>\n",
       "<polyline clip-path=\"url(#clip6702)\" style=\"stroke:#e26f46; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  243.444,1022.67 247.548,1031.94 251.652,1041.07 255.756,1050.06 259.86,1058.92 263.964,1067.64 268.068,1076.23 272.172,1084.69 276.276,1093.01 280.38,1101.21 \n",
       "  284.484,1109.27 288.588,1117.2 292.692,1125 296.796,1132.67 300.9,1140.21 305.004,1147.62 309.108,1154.91 313.212,1162.07 317.316,1169.1 321.42,1176 \n",
       "  325.523,1182.79 329.627,1189.44 333.731,1195.97 337.835,1202.38 341.939,1208.66 346.043,1214.83 350.147,1220.87 354.251,1226.79 358.355,1232.58 362.459,1238.26 \n",
       "  366.563,1243.82 370.667,1249.26 374.771,1254.58 378.875,1259.79 382.979,1264.88 387.083,1269.85 391.187,1274.7 395.291,1279.44 399.395,1284.07 403.499,1288.58 \n",
       "  407.603,1292.98 411.707,1297.26 415.811,1301.43 419.915,1305.5 424.018,1309.44 428.122,1313.28 432.226,1317.01 436.33,1320.63 440.434,1324.15 444.538,1327.55 \n",
       "  448.642,1330.84 452.746,1334.03 456.85,1337.12 460.954,1340.09 465.058,1342.96 469.162,1345.73 473.266,1348.4 477.37,1350.96 481.474,1353.41 485.578,1355.77 \n",
       "  489.682,1358.02 493.786,1360.17 497.89,1362.23 501.994,1364.18 506.098,1366.03 510.202,1367.79 514.306,1369.45 518.41,1371.01 522.514,1372.47 526.617,1373.84 \n",
       "  530.721,1375.11 534.825,1376.28 538.929,1377.36 543.033,1378.35 547.137,1379.25 551.241,1380.05 555.345,1380.76 559.449,1381.38 563.553,1381.91 567.657,1382.35 \n",
       "  571.761,1382.69 575.865,1382.95 579.969,1383.12 584.073,1383.21 588.177,1383.2 592.281,1383.11 596.385,1382.94 600.489,1382.67 604.593,1382.33 608.697,1381.9 \n",
       "  612.801,1381.38 616.905,1380.78 621.009,1380.1 625.112,1379.34 629.216,1378.5 633.32,1377.57 637.424,1376.57 641.528,1375.49 645.632,1374.32 649.736,1373.08 \n",
       "  653.84,1371.76 657.944,1370.37 662.048,1368.9 666.152,1367.35 670.256,1365.73 674.36,1364.03 678.464,1362.26 682.568,1360.41 686.672,1358.49 690.776,1356.5 \n",
       "  694.88,1354.44 698.984,1352.31 703.088,1350.1 707.192,1347.83 711.296,1345.49 715.4,1343.07 719.504,1340.59 723.607,1338.05 727.711,1335.43 731.815,1332.75 \n",
       "  735.919,1330.01 740.023,1327.19 744.127,1324.32 748.231,1321.38 752.335,1318.37 756.439,1315.31 760.543,1312.18 764.647,1308.99 768.751,1305.74 772.855,1302.43 \n",
       "  776.959,1299.06 781.063,1295.63 785.167,1292.14 789.271,1288.59 793.375,1284.99 797.479,1281.33 801.583,1277.61 805.687,1273.84 809.791,1270.01 813.895,1266.13 \n",
       "  817.999,1262.19 822.102,1258.2 826.206,1254.16 830.31,1250.07 834.414,1245.92 838.518,1241.72 842.622,1237.48 846.726,1233.18 850.83,1228.84 854.934,1224.44 \n",
       "  859.038,1220 863.142,1215.51 867.246,1210.98 871.35,1206.4 875.454,1201.77 879.558,1197.1 883.662,1192.38 887.766,1187.63 891.87,1182.82 895.974,1177.98 \n",
       "  900.078,1173.09 904.182,1168.16 908.286,1163.19 912.39,1158.19 916.494,1153.14 920.597,1148.05 924.701,1142.92 928.805,1137.76 932.909,1132.56 937.013,1127.32 \n",
       "  941.117,1122.05 945.221,1116.74 949.325,1111.4 953.429,1106.02 957.533,1100.61 961.637,1095.17 965.741,1089.69 969.845,1084.19 973.949,1078.65 978.053,1073.08 \n",
       "  982.157,1067.48 986.261,1061.85 990.365,1056.2 994.469,1050.51 998.573,1044.8 1002.68,1039.06 1006.78,1033.29 1010.88,1027.5 1014.99,1021.68 1019.09,1015.84 \n",
       "  1023.2,1009.97 1027.3,1004.09 1031.4,998.174 1035.51,992.24 1039.61,986.285 1043.72,980.308 1047.82,974.311 1051.92,968.294 1056.03,962.258 1060.13,956.204 \n",
       "  1064.24,950.131 1068.34,944.041 1072.44,937.934 1076.55,931.811 1080.65,925.672 1084.76,919.519 1088.86,913.351 1092.96,907.169 1097.07,900.973 1101.17,894.765 \n",
       "  1105.28,888.546 1109.38,882.314 1113.48,876.072 1117.59,869.819 1121.69,863.557 1125.8,857.286 1129.9,851.006 1134,844.719 1138.11,838.424 1142.21,832.123 \n",
       "  1146.32,825.815 1150.42,819.502 1154.52,813.184 1158.63,806.862 1162.73,800.536 1166.83,794.207 1170.94,787.875 1175.04,781.542 1179.15,775.207 1183.25,768.872 \n",
       "  1187.35,762.536 1191.46,756.201 1195.56,749.867 1199.67,743.535 1203.77,737.204 1207.87,730.877 1211.98,724.553 1216.08,718.234 1220.19,711.919 1224.29,705.609 \n",
       "  1228.39,699.305 1232.5,693.008 1236.6,686.717 1240.71,680.434 1244.81,674.16 1248.91,667.894 1253.02,661.638 1257.12,655.391 1261.23,649.156 1265.33,642.931 \n",
       "  1269.43,636.719 1273.54,630.518 1277.64,624.331 1281.75,618.158 1285.85,611.998 1289.95,605.854 1294.06,599.724 1298.16,593.611 1302.27,587.515 1306.37,581.435 \n",
       "  1310.47,575.373 1314.58,569.33 1318.68,563.306 1322.79,557.301 1326.89,551.317 1330.99,545.353 1335.1,539.41 1339.2,533.49 1343.31,527.592 1347.41,521.717 \n",
       "  1351.51,515.866 1355.62,510.039 1359.72,504.238 1363.83,498.461 1367.93,492.711 1372.03,486.988 1376.14,481.292 1380.24,475.624 1384.34,469.984 1388.45,464.374 \n",
       "  1392.55,458.793 1396.66,453.242 1400.76,447.723 1404.86,442.235 1408.97,436.779 1413.07,431.356 1417.18,425.966 1421.28,420.609 1425.38,415.288 1429.49,410.001 \n",
       "  1433.59,404.751 1437.7,399.536 1441.8,394.358 1445.9,389.218 1450.01,384.116 1454.11,379.052 1458.22,374.028 1462.32,369.044 1466.42,364.1 1470.53,359.197 \n",
       "  1474.63,354.336 1478.74,349.517 1482.84,344.74 1486.94,340.008 1491.05,335.319 1495.15,330.675 1499.26,326.076 1503.36,321.523 1507.46,317.016 1511.57,312.557 \n",
       "  1515.67,308.145 1519.78,303.781 1523.88,299.466 1527.98,295.2 1532.09,290.985 1536.19,286.82 1540.3,282.706 1544.4,278.644 1548.5,274.634 1552.61,270.677 \n",
       "  1556.71,266.774 1560.82,262.925 1564.92,259.131 1569.02,255.392 1573.13,251.71 1577.23,248.083 1581.33,244.514 1585.44,241.003 1589.54,237.55 1593.65,234.156 \n",
       "  1597.75,230.821 1601.85,227.547 1605.96,224.333 1610.06,221.181 1614.17,218.09 1618.27,215.062 1622.37,212.097 1626.48,209.196 1630.58,206.36 1634.69,203.588 \n",
       "  1638.79,200.881 1642.89,198.241 1647,195.668 1651.1,193.161 1655.21,190.723 1659.31,188.353 1663.41,186.052 1667.52,183.821 1671.62,181.66 1675.73,179.57 \n",
       "  1679.83,177.552 1683.93,175.606 1688.04,173.732 1692.14,171.931 1696.25,170.205 1700.35,168.552 1704.45,166.975 1708.56,165.474 1712.66,164.049 1716.77,162.7 \n",
       "  1720.87,161.429 1724.97,160.236 1729.08,159.122 1733.18,158.087 1737.29,157.132 1741.39,156.257 1745.49,155.463 1749.6,154.751 1753.7,154.121 1757.81,153.574 \n",
       "  1761.91,153.11 1766.01,152.73 1770.12,152.435 1774.22,152.225 1778.32,152.101 1782.43,152.063 1786.53,152.112 1790.64,152.249 1794.74,152.474 1798.84,152.787 \n",
       "  1802.95,153.19 1807.05,153.684 1811.16,154.267 1815.26,154.942 1819.36,155.708 1823.47,156.567 1827.57,157.519 1831.68,158.564 1835.78,159.704 1839.88,160.938 \n",
       "  1843.99,162.268 1848.09,163.693 1852.2,165.215 1856.3,166.834 1860.4,168.551 1864.51,170.366 1868.61,172.28 1872.72,174.294 1876.82,176.407 1880.92,178.622 \n",
       "  1885.03,180.938 1889.13,183.355 1893.24,185.875 1897.34,188.498 1901.44,191.225 1905.55,194.056 1909.65,196.992 1913.76,200.034 1917.86,203.181 1921.96,206.436 \n",
       "  1926.07,209.797 1930.17,213.266 1934.28,216.844 1938.38,220.531 1942.48,224.328 1946.59,228.234 1950.69,232.252 1954.8,236.381 1958.9,240.622 1963,244.976 \n",
       "  1967.11,249.443 1971.21,254.024 1975.31,258.719 1979.42,263.529 1983.52,268.455 1987.63,273.497 1991.73,278.656 1995.83,283.932 1999.94,289.326 2004.04,294.839 \n",
       "  2008.15,300.471 2012.25,306.223 2016.35,312.095 2020.46,318.089 2024.56,324.203 2028.67,330.44 2032.77,336.8 2036.87,343.283 2040.98,349.89 2045.08,356.622 \n",
       "  2049.19,363.479 2053.29,370.462 2057.39,377.57 2061.5,384.806 2065.6,392.17 2069.71,399.661 2073.81,407.282 2077.91,415.031 2082.02,422.911 2086.12,430.921 \n",
       "  2090.23,439.062 2094.33,447.335 2098.43,455.741 2102.54,464.279 2106.64,472.951 2110.75,481.757 2114.85,490.697 2118.95,499.773 2123.06,508.985 2127.16,518.333 \n",
       "  2131.27,527.818 2135.37,537.441 2139.47,547.203 2143.58,557.103 2147.68,567.143 2151.79,577.322 2155.89,587.643 2159.99,598.104 2164.1,608.708 2168.2,619.454 \n",
       "  2172.3,630.343 2176.41,641.376 2180.51,652.553 2184.62,663.875 2188.72,675.342 2192.82,686.955 2196.93,698.715 2201.03,710.622 2205.14,722.677 2209.24,734.881 \n",
       "  2213.34,747.233 2217.45,759.735 2221.55,772.388 2225.66,785.191 2229.76,798.145 2233.86,811.252 2237.97,824.511 2242.07,837.923 2246.18,851.49 2250.28,865.21 \n",
       "  2254.38,879.086 2258.49,893.117 2262.59,907.304 2266.7,921.648 2270.8,936.15 2274.9,950.809 2279.01,965.627 2283.11,980.604 2287.22,995.741 2291.32,1011.04 \n",
       "  \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6702)\" style=\"stroke:#3da44d; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  243.444,911.149 247.548,923.807 251.652,936.291 255.756,948.604 259.86,960.746 263.964,972.717 268.068,984.518 272.172,996.15 276.276,1007.61 280.38,1018.91 \n",
       "  284.484,1030.04 288.588,1041 292.692,1051.8 296.796,1062.44 300.9,1072.91 305.004,1083.21 309.108,1093.36 313.212,1103.34 317.316,1113.16 321.42,1122.83 \n",
       "  325.523,1132.33 329.627,1141.67 333.731,1150.86 337.835,1159.89 341.939,1168.76 346.043,1177.48 350.147,1186.05 354.251,1194.46 358.355,1202.71 362.459,1210.82 \n",
       "  366.563,1218.77 370.667,1226.57 374.771,1234.22 378.875,1241.72 382.979,1249.07 387.083,1256.28 391.187,1263.33 395.291,1270.25 399.395,1277.01 403.499,1283.63 \n",
       "  407.603,1290.11 411.707,1296.44 415.811,1302.63 419.915,1308.68 424.018,1314.58 428.122,1320.35 432.226,1325.97 436.33,1331.46 440.434,1336.81 444.538,1342.02 \n",
       "  448.642,1347.1 452.746,1352.04 456.85,1356.84 460.954,1361.51 465.058,1366.04 469.162,1370.45 473.266,1374.72 477.37,1378.85 481.474,1382.86 485.578,1386.74 \n",
       "  489.682,1390.49 493.786,1394.11 497.89,1397.6 501.994,1400.97 506.098,1404.21 510.202,1407.32 514.306,1410.31 518.41,1413.18 522.514,1415.92 526.617,1418.54 \n",
       "  530.721,1421.03 534.825,1423.41 538.929,1425.67 543.033,1427.81 547.137,1429.82 551.241,1431.72 555.345,1433.51 559.449,1435.17 563.553,1436.73 567.657,1438.16 \n",
       "  571.761,1439.48 575.865,1440.69 579.969,1441.79 584.073,1442.77 588.177,1443.64 592.281,1444.41 596.385,1445.06 600.489,1445.6 604.593,1446.04 608.697,1446.37 \n",
       "  612.801,1446.59 616.905,1446.7 621.009,1446.71 625.112,1446.62 629.216,1446.42 633.32,1446.12 637.424,1445.72 641.528,1445.21 645.632,1444.61 649.736,1443.9 \n",
       "  653.84,1443.1 657.944,1442.2 662.048,1441.19 666.152,1440.1 670.256,1438.9 674.36,1437.61 678.464,1436.23 682.568,1434.75 686.672,1433.18 690.776,1431.52 \n",
       "  694.88,1429.76 698.984,1427.92 703.088,1425.98 707.192,1423.96 711.296,1421.84 715.4,1419.64 719.504,1417.35 723.607,1414.98 727.711,1412.51 731.815,1409.97 \n",
       "  735.919,1407.34 740.023,1404.62 744.127,1401.83 748.231,1398.95 752.335,1395.99 756.439,1392.95 760.543,1389.83 764.647,1386.63 768.751,1383.36 772.855,1380 \n",
       "  776.959,1376.57 781.063,1373.06 785.167,1369.48 789.271,1365.83 793.375,1362.1 797.479,1358.29 801.583,1354.42 805.687,1350.47 809.791,1346.46 813.895,1342.37 \n",
       "  817.999,1338.21 822.102,1333.99 826.206,1329.7 830.31,1325.34 834.414,1320.91 838.518,1316.42 842.622,1311.87 846.726,1307.25 850.83,1302.57 854.934,1297.82 \n",
       "  859.038,1293.01 863.142,1288.15 867.246,1283.22 871.35,1278.23 875.454,1273.19 879.558,1268.09 883.662,1262.92 887.766,1257.71 891.87,1252.44 895.974,1247.11 \n",
       "  900.078,1241.73 904.182,1236.29 908.286,1230.81 912.39,1225.27 916.494,1219.68 920.597,1214.04 924.701,1208.35 928.805,1202.61 932.909,1196.82 937.013,1190.98 \n",
       "  941.117,1185.1 945.221,1179.18 949.325,1173.2 953.429,1167.19 957.533,1161.13 961.637,1155.02 965.741,1148.88 969.845,1142.69 973.949,1136.46 978.053,1130.2 \n",
       "  982.157,1123.89 986.261,1117.55 990.365,1111.16 994.469,1104.74 998.573,1098.29 1002.68,1091.8 1006.78,1085.27 1010.88,1078.72 1014.99,1072.12 1019.09,1065.5 \n",
       "  1023.2,1058.84 1027.3,1052.16 1031.4,1045.44 1035.51,1038.69 1039.61,1031.92 1043.72,1025.11 1047.82,1018.28 1051.92,1011.43 1056.03,1004.55 1060.13,997.64 \n",
       "  1064.24,990.709 1068.34,983.755 1072.44,976.778 1076.55,969.78 1080.65,962.76 1084.76,955.72 1088.86,948.66 1092.96,941.582 1097.07,934.485 1101.17,927.372 \n",
       "  1105.28,920.241 1109.38,913.095 1113.48,905.934 1117.59,898.758 1121.69,891.569 1125.8,884.367 1129.9,877.152 1134,869.927 1138.11,862.691 1142.21,855.445 \n",
       "  1146.32,848.19 1150.42,840.927 1154.52,833.656 1158.63,826.378 1162.73,819.094 1166.83,811.805 1170.94,804.511 1175.04,797.214 1179.15,789.913 1183.25,782.61 \n",
       "  1187.35,775.306 1191.46,768.001 1195.56,760.695 1199.67,753.391 1203.77,746.088 1207.87,738.787 1211.98,731.488 1216.08,724.194 1220.19,716.904 1224.29,709.619 \n",
       "  1228.39,702.34 1232.5,695.068 1236.6,687.804 1240.71,680.547 1244.81,673.3 1248.91,666.062 1253.02,658.834 1257.12,651.618 1261.23,644.414 1265.33,637.222 \n",
       "  1269.43,630.044 1273.54,622.88 1277.64,615.731 1281.75,608.598 1285.85,601.481 1289.95,594.382 1294.06,587.3 1298.16,580.237 1302.27,573.193 1306.37,566.17 \n",
       "  1310.47,559.168 1314.58,552.187 1318.68,545.229 1322.79,538.294 1326.89,531.383 1330.99,524.497 1335.1,517.636 1339.2,510.802 1343.31,503.994 1347.41,497.214 \n",
       "  1351.51,490.463 1355.62,483.74 1359.72,477.048 1363.83,470.387 1367.93,463.757 1372.03,457.159 1376.14,450.594 1380.24,444.063 1384.34,437.566 1388.45,431.105 \n",
       "  1392.55,424.679 1396.66,418.291 1400.76,411.939 1404.86,405.626 1408.97,399.352 1413.07,393.118 1417.18,386.924 1421.28,380.772 1425.38,374.661 1429.49,368.593 \n",
       "  1433.59,362.569 1437.7,356.589 1441.8,350.653 1445.9,344.764 1450.01,338.921 1454.11,333.125 1458.22,327.378 1462.32,321.679 1466.42,316.029 1470.53,310.43 \n",
       "  1474.63,304.882 1478.74,299.385 1482.84,293.941 1486.94,288.551 1491.05,283.214 1495.15,277.932 1499.26,272.706 1503.36,267.535 1507.46,262.422 1511.57,257.367 \n",
       "  1515.67,252.37 1519.78,247.432 1523.88,242.554 1527.98,237.737 1532.09,232.982 1536.19,228.289 1540.3,223.659 1544.4,219.092 1548.5,214.59 1552.61,210.154 \n",
       "  1556.71,205.783 1560.82,201.479 1564.92,197.243 1569.02,193.075 1573.13,188.976 1577.23,184.946 1581.33,180.988 1585.44,177.1 1589.54,173.285 1593.65,169.542 \n",
       "  1597.75,165.873 1601.85,162.278 1605.96,158.758 1610.06,155.314 1614.17,151.946 1618.27,148.656 1622.37,145.444 1626.48,142.31 1630.58,139.256 1634.69,136.282 \n",
       "  1638.79,133.39 1642.89,130.579 1647,127.85 1651.1,125.205 1655.21,122.644 1659.31,120.168 1663.41,117.777 1667.52,115.473 1671.62,113.256 1675.73,111.126 \n",
       "  1679.83,109.085 1683.93,107.133 1688.04,105.271 1692.14,103.501 1696.25,101.821 1700.35,100.234 1704.45,98.74 1708.56,97.3397 1712.66,96.034 1716.77,94.8236 \n",
       "  1720.87,93.7093 1724.97,92.6917 1729.08,91.7717 1733.18,90.95 1737.29,90.2273 1741.39,89.6045 1745.49,89.0822 1749.6,88.6612 1753.7,88.3423 1757.81,88.1262 \n",
       "  1761.91,88.0136 1766.01,88.0053 1770.12,88.1021 1774.22,88.3048 1778.32,88.6139 1782.43,89.0304 1786.53,89.555 1790.64,90.1883 1794.74,90.9313 1798.84,91.7845 \n",
       "  1802.95,92.7488 1807.05,93.825 1811.16,95.0137 1815.26,96.3157 1819.36,97.7318 1823.47,99.2627 1827.57,100.909 1831.68,102.672 1835.78,104.552 1839.88,106.55 \n",
       "  1843.99,108.666 1848.09,110.901 1852.2,113.257 1856.3,115.734 1860.4,118.332 1864.51,121.052 1868.61,123.896 1872.72,126.863 1876.82,129.955 1880.92,133.172 \n",
       "  1885.03,136.516 1889.13,139.986 1893.24,143.584 1897.34,147.31 1901.44,151.165 1905.55,155.15 1909.65,159.265 1913.76,163.512 1917.86,167.892 1921.96,172.403 \n",
       "  1926.07,177.049 1930.17,181.829 1934.28,186.744 1938.38,191.794 1942.48,196.982 1946.59,202.307 1950.69,207.769 1954.8,213.371 1958.9,219.112 1963,224.994 \n",
       "  1967.11,231.017 1971.21,237.181 1975.31,243.489 1979.42,249.939 1983.52,256.534 1987.63,263.273 1991.73,270.158 1995.83,277.19 1999.94,284.368 2004.04,291.695 \n",
       "  2008.15,299.17 2012.25,306.794 2016.35,314.569 2020.46,322.494 2024.56,330.571 2028.67,338.801 2032.77,347.183 2036.87,355.72 2040.98,364.411 2045.08,373.257 \n",
       "  2049.19,382.26 2053.29,391.419 2057.39,400.736 2061.5,410.211 2065.6,419.846 2069.71,429.64 2073.81,439.595 2077.91,449.711 2082.02,459.99 2086.12,470.431 \n",
       "  2090.23,481.036 2094.33,491.806 2098.43,502.74 2102.54,513.841 2106.64,525.108 2110.75,536.542 2114.85,548.144 2118.95,559.916 2123.06,571.857 2127.16,583.968 \n",
       "  2131.27,596.251 2135.37,608.706 2139.47,621.333 2143.58,634.133 2147.68,647.108 2151.79,660.257 2155.89,673.583 2159.99,687.084 2164.1,700.763 2168.2,714.62 \n",
       "  2172.3,728.655 2176.41,742.87 2180.51,757.264 2184.62,771.84 2188.72,786.597 2192.82,801.537 2196.93,816.66 2201.03,831.967 2205.14,847.459 2209.24,863.135 \n",
       "  2213.34,878.999 2217.45,895.049 2221.55,911.286 2225.66,927.712 2229.76,944.327 2233.86,961.133 2237.97,978.128 2242.07,995.315 2246.18,1012.69 2250.28,1030.27 \n",
       "  2254.38,1048.03 2258.49,1065.99 2262.59,1084.15 2266.7,1102.5 2270.8,1121.05 2274.9,1139.79 2279.01,1158.74 2283.11,1177.88 2287.22,1197.22 2291.32,1216.77 \n",
       "  \n",
       "  \"/>\n",
       "<path clip-path=\"url(#clip6700)\" d=\"\n",
       "M254.008 312.204 L702.668 312.204 L702.668 130.764 L254.008 130.764  Z\n",
       "  \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
       "<polyline clip-path=\"url(#clip6700)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  254.008,312.204 702.668,312.204 702.668,130.764 254.008,130.764 254.008,312.204 \n",
       "  \"/>\n",
       "<polyline clip-path=\"url(#clip6700)\" style=\"stroke:#e26f46; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  278.008,191.244 422.008,191.244 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip6700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 446.008, 208.744)\" x=\"446.008\" y=\"208.744\">LS</text>\n",
       "</g>\n",
       "<polyline clip-path=\"url(#clip6700)\" style=\"stroke:#3da44d; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
       "  278.008,251.724 422.008,251.724 \n",
       "  \"/>\n",
       "<g clip-path=\"url(#clip6700)\">\n",
       "<text style=\"fill:#000000; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:48px; text-anchor:start;\" transform=\"rotate(0, 446.008, 269.224)\" x=\"446.008\" y=\"269.224\">Orth. dist.</text>\n",
       "</g>\n",
       "</svg>\n"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "using Plots\n",
    "scatter(xd,yd, label=\"\", legend = :topleft);\n",
    "x = linspace(minimum(xd), maximum(xd), 500);\n",
    "y_ls = vandermonde(x, p) * theta_ls;\n",
    "y_od = vandermonde(x, p) * theta_od;\n",
    "plot!(x, y_ls, label = \"LS\")\n",
    "plot!(x, y_od, label = \"Orth. dist.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Figure 18.<a ind=\"#fig6\"></a>6** Least squares and orthogonal distance regression fit of a cubic polynomial to $25$ data points."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 1.1.1",
   "language": "julia",
   "name": "julia-1.1"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.1.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
